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ABSTRACT 


Earning  a  college  degree  is  an  aspiration  of  many,  and  on-line  distance  learning 
(DL)  is  a  feasible  way  to  attain  that  level  of  education.  The  Naval  Postgraduate 
School  (NPS)  offers  masters-  and  doctorate-level  degrees  to  federal  government 
employees  via  resident  and  DL  means.  Does  either  method  of  delivery  provide  a 
better,  or  worse,  opportunity  for  strong  student  performance?  Do  available 
student  characteristics  lead  to  better  performance  in  one  method  or  the  other? 

This  study  analyzed  the  performance  of  2,633  student  Navy  officers  in  the 
NPS  Graduate  School  of  Business  and  Public  Policy  (GSBPP),  the  Graduate 
School  of  Engineering  and  Applied  Science  (GSEAS)  and  the  Graduate  School 
of  Operational  and  Information  Science  (GSOIS)  in  the  DL  and  resident  formats. 
The  analysis  used  simple  linear  models,  general  linear  models,  and  recursive 
partitioning  to  determine  which  of  ten-selected  predictors  can  identify  strong  or 
poor  student  performance.  Results  of  the  analysis  showed  the  NPS  Academic 
Profile  Code  (APC)  is  a  strong  indicator  of  an  increased  probability  of  success, 
while  DL  students  in  GSEAS  and  GSOIS  are  at  greatest  risk  of  poor 
performance.  More  research  is  recommended  to  determine  why  those  students 
have  difficulty  succeeding  at  NPS. 
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EXECUTIVE  SUMMARY 


College  education  via  distance  learning  (DL)  in  an  online  setting  is  an 
education  path  taken  by  millions  of  students  and  offered  by  thousands  of 
colleges.  The  Naval  Postgraduate  School  (NPS)  has  been  offering  DL  courses  to 
federal  government  employees  for  over  two  decades.  However,  unlike  many 
civilian  institutions  of  higher  education,  NPS  has  yet  to  conduct  a  comprehensive, 
full  university  analysis  to  determine  if  students  enrolled  through  its  DL  program 
do  as  well  as  their  resident-student  counterparts.  This  is  the  first  study  to  do  so 
by  analyzing  the  performance  of  DL  and  resident  Navy  officer  students  at  NPS. 
In  addition,  NPS’s  primary  admissions  tool,  the  Academic  Profile  Code  (APC) 
was  examined  to  determine  its  ability  to  predict  student  success. 

This  study  is  an  investigation  of  the  performance  of  2,633  Navy  officers 
who  enrolled  in  DL  and  resident  programs  in  the  Graduate  School  of  Business 
and  Public  Policy  (GSBPP),  Graduate  School  of  Engineering  and  Applied 
Sciences  (GSEAS),  and  the  Graduate  School  of  Operational  and  Information 
Sciences  (GSOIS)  at  NPS  from  academic  years  2006  to  2014.  Student  success 
was  defined  in  two  ways:  by  the  student  meeting  all  requirements  for  graduation 
from  NPS  (a  Graduate  Quality  Point  Rating  [GQPR]  of  3.0  or  better)  and 
performing  well  enough  to  be  selected  for  graduation  “with  distinction.”  The 
opposite  of  student  success,  disenrollment  from  NPS  for  either  academic  or 
administrative  purposes,  was  also  analyzed  to  develop  a  thorough  understanding 
of  both  ends  of  the  student-performance  spectrum. 

This  exploration  of  student  performance  was  done  by  selecting  ten 
probable  predictors  that  can  determine  a  student’s  Total  Quality  Point  Rating 
(TQPR),  graduation  eligibility,  ability  to  graduate  “with  distinction,”  enrollment 
status,  and  TQPR  of  disenrolled  students  for  a  total  of  five  possible  response 
variables.  Simple  linear  models  (LMs)  and  recursive  partition  (RP)  regression 
trees  (RTs)  were  developed  to  analyze  the  continuous  response  variables  of 

TQPR  (0.0  to  4.0).  General  linear  models  (GLM)  with  a  logit  link  and  RP 

xvii 


classification  trees  (CT)  were  developed  to  analyze  the  response  variables  with  a 
yes  or  no  value.  In  all,  a  grand  total  of  ten  separate  models  were  created  to 
develop  a  thorough  understanding  of  DL  and  resident  student  performance  at 
NPS. 

Results  of  the  two  LMs,  three  GLMs,  two  RTs,  and  three  CTs  provided 
intriguing  insight  into  all  of  the  variables  used  as  predictors.  Of  the  two  main 
predictors  of  interest,  DL  was  prominent  in  seven  of  the  ten  models,  while  ARC 
was  important  in  five  of  the  ten  models.  Notably  DL  holds  no  influence  in 
determining  high  student  performance  (graduating  “with  distinction”),  while  ARC 
is  extremely  influential.  Conversely,  DL  is  a  major  determinant  of  poor  student 
performance  (disenrollment  for  NPS),  while  APC  holds  minimal  sway  in 
determining  student  failure. 

In  addition  to  APC,  only  a  few  other  variables  were  singled  out  as  good 
signals  of  student  success:  time  between  undergraduate  degree  and  NPS 
attendance  (14  years  or  less),  and  United  States  Naval  Academy  (USNA) 
graduate  status  (Yes).  Aside  from  these  three  predictors,  there  are  very  few 
distinguishable  characteristics  among  successful  students  at  NPS.  In  contrast,  a 
theme  developed  identifying  poor  student  performance:  Navy  Officers  taking  DL 
courses  in  GSEAS  and  GSOIS  were  a  noticeable  majority  of  those  who  were 
disenrolled  from  NPS. 

It  is  too  early  to  determine  the  exact  reason  for  this  population’s  lack  of 

student  success.  One  possible  explanation  is  that  the  students  obligate  to  a  DL 

program  without  a  full  understanding  of  the  time  commitment  required  for 

success.  Another  explanation  is  possible  inconsistencies  in  the  delivery  of  the 

coursework  via  DL  due  to  the  instructor’s  lack  of  familiarity  with  the  format.  It  may 

also  be  possible  that  students  and  instructors  are  fully  prepared  for  the  DL 

experience,  but  the  high  level  of  comprehension  required  for  the  technical 

curricula  offered  by  both  GSEAS  and  GSOIS  make  it  very  difficult  for  the 

knowledge  to  be  transferred  effectively  via  the  DL  medium.  Follow  on  work  is 

necessary  to  pinpoint  the  source  of  this  shortfall. 
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I.  INTRODUCTION 


Receiving  a  graduate  education  is  a  strong  indicator  of  an  individual’s 
personal  motivation  and  professional  potential.  Earning  a  master’s  degree  can  be 
considered  an  unofficial  “check  in  the  box”  to  many,  but,  for  most  military  officers, 
doing  so  also  improves  promotion  opportunities  to  the  next  higher  paygrade. 
Unfortunately,  the  rigid  career  path  of  most  Navy  officer  communities  does  not 
allow  for  the  opportunity  to  step  away  for  18  months  or  more  to  attend  the 
resident  program  at,  for  example,  the  Naval  Postgraduate  School  (NPS). 
Fortunately  for  these  officers,  NPS  has  been  a  leader  in  providing  distance 
learning  (DL)  education  to  eligible  learners  for  nearly  two  decades  by  providing 
resources  such  as  Elluminate/Collaborate,  Video  Teleconferencing  (VTC),  hybrid 
programs,  and  more. 

In  addition  to  NPS,  many  civilian  institutions  have  developed  vast  DL 
programs  and,  within  recent  years,  DL  has  become  a  substantial  source  of 
income  for  these  civilian  institutions  (Shachar  &  Neumann,  2010).  With  a  new 
source  of  increased  income  comes  a  desire  to  optimize  cash  flow  from  that 
source;  therefore  DL  offerings  have  become  more  robust  (through  more  classes 
offered,  more  availability,  more  resources  committed,  and  more  opportunities  for 
degrees,  all  via  DL)  throughout  academia  (Shachar  &  Neumann,  2010).  Rather 
than  grow  DL  programs  haphazardly,  civilian  institutions  have  performed  their 
due  diligence  to  determine  whether  or  not  DL  curricula  do,  in  fact,  provide  an 
education  equivalent  to  a  resident  student’s  education.  Many  institutions  have 
published  studies  that  explore  the  effectiveness  of  DL  (Means,  Toyama,  Murphy, 
&  Baki,  2013). 

Today  the  opportunities  for  professionals  to  pursue  advanced  education 
concurrently  with  their  careers  are  nearly  limitless  via  DL,  and  navy  leadership  is 
very  interested  in  expanding  DL  opportunities  for  Navy  officers.  NPS  has 
provided  DL  opportunities  for  nearly  two  decades  (Barrett,  1996)  but  their 

comparability  to  resident  education  has  been  only  loosely  monitored.  For 
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example,  in  2012,  the  NPS  Office  of  Institutional  Research  and  Planning  (OIRP) 
released  an  informational  newsletter  detailing  the  disparity  in  graduation  rates  for 
the  Fiscal  Year  2002-2008  cohorts  (2012).  The  Graduate  School  of  Engineering 
and  Applied  Sciences  maintains  a  database  of  its  DL  and  resident  student 
performance.  Aside  from  descriptive  statistics  and  anecdotal  information,  an 
NPS-wide  data  analysis  comparing  the  effectiveness  of  an  NPS  DL  education  to 
NPS  resident  education  has  yet  to  be  completed.  Before  moving  forward  with 
expanding  DL  programs  at  NPS,  it  is  prudent  to  determine  empirically  if  those 
who  have  earned  NPS  degrees  via  DL  are  just  as  successful  in  their  studies  as 
their  resident  counterparts. 

A.  DISTANCE  LEARNING  COURSES  AT  NPS 

Naval  Postgraduate  School  (2014)  offers  DL  curricula  within  three  of  its 
four  separate  schools.  The  Graduate  School  of  Business  and  Public  Policy 
(GSBPP)  offers  Executive  Master  of  Business  Arts  (MBA)  and  Program  and 
Contract  Management  DL  degrees.  The  Graduate  School  of  Engineering  and 
Applied  Sciences  (GSEAS)  administers  degree  programs  in  Space  Systems, 
Electrical,  and  Mechanical  Engineering,  among  others.  Degrees  in  Systems 
Analysis  and  Computer  Science  are  offered  through  the  Graduate  School  of 
Operational  and  Information  Sciences  (GSOIS). 

B.  THE  ACADEMIC  PROFILE  CODE 

In  addition  to  analyzing  any  possible  differences  in  student  achievement 
between  DL  and  resident  programs,  we  also  looked  closely  at  the  NPS  Academic 
Profile  Code  (APC)  and  its  connection  to  student  success  in  both  DL  and 
resident  programs.  Used  as  the  prime  screening  tool  for  prospective  NPS 
graduate  students,  the  APC  is  based  on  each  applicant’s  overall  undergraduate 
performance  (graduating  grade  point  average  [GPA]),  and  performance  in  upper- 
level  calculus  and  calculus-based  physics  courses. 

Nearly  all  curricula  at  NPS  have  a  required  APC  for  admission.  Per  the 

NPS  Academic  Catalog  (2014)  the  APC  is  a  three-digit  code  that  provides  details 

2 


on  each  student’s  success  as  an  undergraduate  (based  on  transcripts)  and  their 
projected  propensities  for  success  in  the  respective  curriculum.  Each  digit  of  a 
student’s  assigned  ARC  must  be  less  than  or  equal  to  the  required  ARC  for  his  or 
her  curriculum  of  study.  In  cases  where  a  prospective  student’s  ARC  digit  does 
not  meet  that  requirement,  the  respective  department  chair  (which  can  be 
delegated  to  Rrogram  Officers  and  Academic  Associates)  can  provide  a  waiver  to 
allow  a  student  into  the  desired  program. 

1.  ARC  First  Digit 

The  first  digit  of  an  ARC  indicates  overall  academic  performance  based  on 
a  recalculated  GRA.  It  incorporates  failures  and  repeated  courses  from  all 
previous  college  transcripts.  This  first  digit  is  derived  from  the  information  in 
Table  1  : 

Table  1 .  ARC  First  Digit  code  designation  by  undergraduate  GRA 
(after  Naval  Rostgraduate  School,  2014a). 


Code 

GPA  Range 

0 

3.604,00 

1 

3.20-3.59 

2 

2.60-3.19 

3 

2.20-2.59 

4 

1.90-2.19 

5 

0.00-1.89 

According  to  the  Naval  Rostgraduate  School  course  catalogue  (2014a),  a 
first  digit  code  of  0,  1,  2  or  3  (as  appropriate)  is  assigned  only  if  transcripts 
provided  exhibit  at  least  75  semester-hours  or  1 12  quarter-hours  of  actual  graded 
classroom  instruction.  Grades  of  Rass/Fail  and  Credit/No  Credit  do  not  count 
toward  the  75/112-hour  requirement. 
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2.  APC  Second  Digit 

The  second  digit  represents  the  student’s  mathematical  background.  All 
math  courses  from  calculus  through  post-calculus  are  considered  when 
evaluating  the  transcripts  for  the  second  digit.  A  minimum  calculus  sequence  is 
Calculus  I  and  II.  Possible  values  are  shown  in  Table  2. 


Table  2.  APC  Second  Digit  code  designation  by  undergraduate  math 
experience  (after  Naval  Postgraduate  School,  2014a). 


Code 

Meaning 

0 

Math  Major/Minor,  Quantitative  Economics  Degree  with  B  or  better 
average;  math  taken  less  than  or  equal  to  7  years  ago. 

1 

Lower  Level,  Upper  Level,  Linear  Algebra  with  a  GPA  of  at  least  a  3.5; 
math  taken  less  than  or  equal  to  5  years  ago. 

2 

Lower  Level,  Upper  Level  with  average  between  C+  and  B+,  math  taken 
less  than  or  equal  to  5  years  ago.  No  Linear  Algebra. 

3 

Lower  Level  Calculus  Sequence  with  a  C  or  better;  or  if  math  taken 
greater  than  5  years  ago. 

4 

Calculus  for  Business/Sodal  Sciences  with  a  C  or  better.  1  Lower  Level 
Calculus  Course  with  at  least  a  C-.  2  pre-Calculus  Courses 
with  a  B+  or  better. 

5 

At  least  one  pre-Calculus  with  C-  or  better  grade. 

6 

No  pertinent  college-level  math  with  a  grade  of  C-  or  better. 

4 


3.  APC  Third  Digit 

The  third  digit  represents  previous  course  coverage  in  science  and 
technical  fields  according  to  the  criteria  in  Table  3. 


Table  3.  APC  Third  Digit  code  designation  by  physics  experience 
(after  Naval  Postgraduate  School,  2014a). 


Code 

Eng/Tech  GPA 

Meaning 

0 

3.00-4,00 

Accreditation  Board  for  Engineering  and  Technology 
(ABET)  Engineering  Accreditation 
Commission  (EAC)  accredited,  B.S, 
Engineering  Degree  (regardless  of  time 
passed) 

1 

>  =  2.30 

Non-ABET  EAC  accredited,  Engineering  Degree 
(regardless  of  time  passed) 

2 

>  =  2.30 

Any  B.S.  Technical  degree  (regardless  oftime  passed) 

3 

>  =  3.00 

Completed  calculus-based  physics  sequence  with  a  B 
average  or  above 

4 

>  =  2.00 

One  calculus-based  physics  course  with  at  least  a  C 

5 

<  =  1.99 

No  pertinent  technical  courses. 

a.  Engineering  Degrees 

Engineering  degrees  include  Aeronautical,  Computer,  Electrical, 
Mechanical,  Materials,  Marine,  Naval,  Ocean,  Systems,  Industrial,  Chemical, 
Bioengineering,  and  Naval  Architecture. 

b.  Technicai  Degrees 

Technical  degrees  include  Applied  Physics,  Engineering  Physics,  and 
Physics. 


c.  Generai  Engineering  and  Eiectricai/Mechanicai  Engineering 
Technoiogy 

These  degrees  are  not  counted  as  engineering  degrees  or  technical 
degrees  for  the  purposes  of  calculating  an  APC. 
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d.  APC  Code  Requirements 

When  calculating  the  APC,  if  the  record  cannot  meet  all  the  requirements 
to  obtain  Code  0  (i.e.,  GPA  is  2.75  but  all  other  requirements  are  met)  the  Code 
drops  to  a  1  automatically  but  no  further. 

A  discussion  with  the  NPS  Director  of  Admissions,  LtCol(Ret)  Susan 
Dooley,  revealed  that  the  actual  genesis  of  the  APC  as  an  admissions  tool  is  not 
documented.  The  oldest  information  she  has  on  the  APC  dates  back  to  1983.  In 
addition,  APC  determination  requirements  for  the  second  and  third  digit  were 
adjusted  during  the  time  span  of  the  collection  of  the  data  being  analyzed.  This 
was  done  to  better  reflect  an  applicant’s  education  and  preparation  for  the  level 
of  mathematic  and  technical  rigor  in  the  programs  at  NPS. 

C.  SCOPE  OF  THIS  THESIS 

The  purpose  of  this  thesis  study  is  to  examine  NPS  entrance  requirements 
and  subsequent  student  performance  of  U.S.  Navy  officers  to  ultimately 
determine  if  NPS  DL  programs  are  as  effective  as  resident  programs.  The  goal  is 
to  understand  predictors  of  student  success  (determined  by  looking  at  student 
attrition,  all  graduating  Quality  Point  Ratings  [QPRs],  and  those  who  graduated 
“with  distinction”)  in  both  DL  and  resident  programs.  With  this,  we  intend  to  build 
on  the  understanding  of  how  NPS  can  best  ensure  the  success  of  its  students  in 
either  program,  and  how  NPS  can  expand  its  DL  offerings. 

D.  RESEARCH  QUESTIONS 

1 .  Is  the  NPS  APC  a  valid  predictor  of  student  success  in  both  DL  and 
resident  programs? 

2.  Do  graduate  students  achieve  a  higher  level  of  student 
performance  in  a  resident  education  or  in  a  distance  learning 
education? 

3.  What  student  attributes  lead  to  success  in  distance  learning  versus 
resident  learning  (and  vice  versa),  and  how  they  differ? 
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E.  ORGANIZATION  OF  THIS  THESIS 

This  thesis  contains  five  chapters.  Chapter  I  provides  a  general  overview 
and  background  on  the  area  of  analysis.  Chapter  II  provides  literature  and 
studies  relating  to  DL  and  resident  education  and  how  they  contrast.  Chapter  III 
discusses  the  variables  used  in  the  regression  and  classification  models  for  all 
active  and  reserve  component  Navy  officers  (and  a  few  enlisted  personnel)  that 
studied  through  NPS  during  the  academic  years  2006  through  2014.  Chapter  IV 
gives  the  details  of  building  regression  and  classification  models  to  determine  if 
there  is  a  difference  in  student  performance  between  DL  and  Resident  students 
and  whether  or  not  the  NPS  Academic  Profile  Code  is  a  valid  predictor  of 
success  at  NPS.  Chapter  V  offers  conclusions,  recommendations  for  further 
research  and  recommendations  to  improve  the  DL  program  at  NPS. 
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II.  LITERATURE  REVIEW 


If  one  talks  to  most  Americans  today,  the  term  “Distance  Learning”  will 
invoke  a  schema  of  VTCs  and  online  courses  through  a  university  located  in 
some  other  part  of  the  country,  if  there  even  is  an  existing  campus  (Bowers  & 
Kumar,  2015).  It  is  easy  to  assume  that  DL  programs  have  only  existed  since  the 
mid-1990s  when  the  available  technology  allowed  for  instructors  and  students  to 
collaborate  online  via  email  and  message  boards.  Many  forget  that  a  distance 
learning  class  can  also  be  referred  to  as  a  correspondence  course.  In  fact,  the 
first  university  to  offer  any  distance  learning  opportunity  is  the  University  of 
London  when  it  established  the  University  of  London  International  Programmes 
in  1858  (University  of  London,  2014). 

A.  DISTANCE  LEARNING  ATTRITION 

The  ability  of  a  student  to  maintain  his  or  her  studies  via  a  Distance 

Learning  program  is  a  well-known  challenge  to  educators  involved  in  DL.  It  is 

considered  easier  for  DL  students  to  drop  out  because  they  are  normally  not 

located  at  the  school.  In  addition,  for  Navy  officers,  being  a  student  is  the  primary 
occupation  for  those  in  resident  status.  For  those  taking  courses  via  DL,  it  is  a 
duty  secondary  to  the  student’s  primary  billet  at  a  Navy  command.  Dropping  out 
of  NPS  as  a  Resident  requires  new  orders  for  a  Permanent  Change  of  Station 
(PCS),  likely  to  another  geographic  location  (since  there  are  very  few  active  duty 
billets  located  in  the  Monterey,  CA  area  outside  of  NPS).  This  is  a  large, 
unplanned  cost  to  the  Navy’s  detailing  distribution  funds.  Dropping  out  of  an  NPS 
DL  program  means  only  that  the  Navy  officer  actually  has  one  less  responsibility 
to  complete  in  addition  to  already  demanding  duties.  Since  NPS  DL  classes  are 
taken  at  no  fiscal  cost  to  the  student  Navy  officer  (they  do,  however,  incur 
required  service  for  courses  attempted),  dropping  classes  does  not  lead  to  the 
burden  of  personal  funds  wasted  (which  DL  students  experience  when  dropping 
classes  through  civilian  institutions). 
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1. 


NPS  Research 


Studies  on  attrition  from  NPS  (students  disenrolled  for  academic  and/or 
administrative  reasons)  are  virtually  non-existent.  Therefore,  it  is  no  surprise  that 
there  have  been  no  studies  on  DL  attrites  from  NPS.  Aside  from  this  fact,  the 
studies  on  attrition  from  enlisted  training  programs,  the  Naval  Academy,  Naval 
Reserve  Officer  Training  Corps  (NROTC)  programs,  and  officer  training 
programs  (such  as  flight  school)  are  numerous  and  a  handful  of  recent  studies  at 
NPS  have  analyzed  student  attrition  from  civilian  DL  programs  via  the  Navy 
College  Program  for  Afloat  College  Education  (NCPACE)  and  tuition  assistance 
(TA). 


a.  Navy  College  Program  for  Afloat  College  Education 

NCPACE  is  a  tuition-free  education  opportunity  available  to  forward- 
deployed  sailors  in  operational  environments  offered  through  a  handful  of 
academic  institutions  in  agreement  with  the  U.S.  Navy.  There  are  two  delivery 
options  available:  classroom  instruction  where  the  teacher  deploys  with  a  large 
unit,  such  as  an  aircraft  carrier,  or  the  DL  model  for  those  sailors  deployed  with 
smaller  units  (Chief  of  Naval  Education  and  Training,  2014).  Park  (2011) 
analyzed  206,803  NCPACE  courses  taken  between  fiscal  year  (FY)  1995  and 
2008  to  see  how  well  these  deployed  sailors  complete  and  perform  in  the 
courses  taken  via  classroom  and  DL  models  (and  how  well  these  students 
succeeded  in  their  respective  careers  after  course  completion,  but  that  is  outside 
the  scope  of  this  study).  Through  her  analysis,  she  discovered  that  students 
taking  a  course  through  classroom  instruction  were  ten  times  more  likely  to 
complete  the  course  than  those  sailors  taking  a  course  via  the  DL  model. 

b.  Tuition  Assistance 

TA  is  another  tool  available  to  sailors  (in  fact,  available  to  all  service- 

members)  to  advance  their  own  educations.  This  differs  from  NCPACE  because 

each  member  using  TA  is  able  to  register  for  courses  at  any  approved  accredited 

institution  and  request  TA  funds  to  cover  the  cost  of  tuition.  TA  can  also  be  used 

10 


at  any  time  by  any  service  member;  NCPACE  can  only  be  used  while  the  service 
member  is  in  a  forward  deployed  status.  McLaughlin  (2010)  looked  at  the  course 
performance  of  233,459  sailors  using  TA  to  pay  for  tuition  from  FY  1994  to  2008. 
His  analysis  revealed  that  sailors  enrolled  in  DL  courses  are  10.7%  points  less 
likely  to  successfully  complete  a  course  than  sailors  enrolled  in  resident  courses. 
Mehay  and  Pema  (2010)  analyzed  the  same  data  set  and  discovered  similar 
results.  By  using  fixed  effects  (holding  all  but  one  predictor  constant  while 
adjusting  that  one  predictor  to  see  how  it  affects  the  respondent),  they  found  an 
8%  lower  pass  rate  in  DL  courses  than  in  resident  courses. 

2.  Civilian  Research 

Attrition  from  DL  courses  of  civilian  institutions  is  a  major  concern  to  their 
respective  administrations.  Completion  rates  are  an  important  measure  of 
success  in  higher  education  because  funding  and  accreditation  are  closely  tied  to 
enrollment  and  course  quality,  respectively  (Howell,  Laws,  &  Lindsay,  2004). 
With  statistics  showing  attrition  in  DL  courses  10-20%  higher  than  resident 
courses  (Holder,  2007);  studies  have  been  conducted  to  determine  why  DL 
student  attrition  is  consistently  greater  than  resident  student  attrition. 

a.  Howell,  Laws,  and  Lindsay 

From  the  start,  Howell,  Laws,  and  Lindsay  (2004)  argue  that  any  research 
comparing  DL  students  to  resident  students  is  erroneous,  the  classic  apples  to 
oranges  comparison.  They  provided  seven  situational  factors  that  DL  students 
have  in  contrast  to  their  resident  counterparts: 

1 .  Delayed  college  enrollment 

2.  Hold  a  Graduate  Equivalency  Diploma 

3.  Financial  independence 

4.  Have  children 

5.  Single  parent 
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6.  Part-time  college  student 

7.  Full-time  worker  during  college 

According  to  Howell,  et  al.  students  with  these  characteristics  (who  are 
unable  to  attend  the  traditional  classroom)  make  up  a  separate  population  from 
resident  students  and,  because  of  this,  DL  student  samples  should  only  be 
compared  to  that  of  other  DL  student  samples,  from  either  other  schools  or 
previous  cohorts. 

b.  Street 

Street  (2010)  reviews  a  handful  of  research  papers  that  attempted  to 
isolate  factors  that  lead  DL  students  to  decide  to  not  complete  a  course.  Street 
discovered  a  total  of  nine  factors  that  lead  to  this  decision  and  grouped  them  into 
three  major  factors: 

1 .  Course  Factors 

•  Relevance 

•  Design 

2.  Environmental  Factors 

•  Family  Support 

•  Organizational  Support 

•  Technical  Support 

3.  Person  Factors 

•  Self-Efficacy 

•  Self  Determination 

•  Autonomy 

•  Time  Management  Skills 
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B.  DISTANCE  LEARNING  EFFECTIVENESS 


1.  NFS  Research 

Fortunately  two  of  the  three  previously  cited  NPS  studies  that  analyzed 
attrition  from  DL  courses  compared  to  resident  courses  also  studied  the 
achievement  levels  of  students  who  did  finish  their  courses  and  received  a  grade. 

a.  McLaughlin 

McLaughlin  (2010)  conducted  a  study  of  sailors  using  TA  to  further  their 
respective  educations.  He  found  that  DL  students,  on  average,  achieved  a  half 
letter  grade  lower  than  their  resident  counterparts. 

b.  Mehay  and  Pema 

With  a  deeper  look  into  the  same  data,  Mehay  and  Pema  (2010)  saw  a 
slightly  smaller  effect  in  the  DL  versus  resident  student  scores.  “Since  the 
average  grade  in  the  sample  is  3.18,  or  slightly  above  a  B,  taking  an  online  class 
reduces  this  to  a  2.92,  or  slightly  below  a  B”  (Mehay  &  Pema,  2010).  In  addition, 
they  also  found  that  pass  rates  in  DL  History  and  English  courses  were  lower 
than  for  resident  courses  in  the  same  disciplines. 

2.  Civilian  Research 

Civilian  institutions  of  higher  education  have  conducted  an  immense 
amount  of  research  looking  at  how  well  DL  students  perform  compared  to  their 
resident  counterparts.  Since  the  online  facet  of  DL  has  existed  in  some  form  for 
25  to  30  years,  a  couple  of  thousand  studies  on  this  very  subject  exist.  These 
many  studies  provide  other  researchers  with  the  opportunity  to  conduct  meta¬ 
analyses  to  analyze  the  trends  and  provide  a  thorough  overview  to  the  rest  of 
academia. 
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a.  Shachar  and  Neumann 

Shachar  and  Neumann  (2010)  conducted  a  meta-analysis  by  reviewing 
1,850  comparative  studies  (conducted  between  1990  and  2010)  and,  through  a 
very  rigorous  filtering  process,  arrived  at  an  analysis  of  125  qualifying  studies 
with  a  total  of  20,800  participating  students.  Of  these  125  studies,  87  of  them 
(70%)  showed  that  DL  had  an  overall  positive  effect.  Based  on  this  (and  a  Chi- 
Square  (df=1)  of  32.13,  (p<0.0001))  the  researchers  confidently  declared  that  the 
DL  students  outperformed  (based  on  grades)  their  resident  counterparts  “across 
the  full  continuum  of  the  study  period.”  In  closing,  Shachar  and  Neumann  boldly 
stated  “the  paradigm  of  the  superiority  of  the  [resident]  modality  over  its  distance 
learning  alternative  has  been  successfully  negated”  (2010). 

b.  Means,  Toyama,  Murphy,  and  Baki 

Even  though  Shachar  and  Neumann  (2013)  claimed  ultimate  victory  for 
the  DL  education  over  the  resident  education  model,  the  argument  is  not  settled. 
Means  et  al.  (2013)  analyzed  99  (out  of  an  initial  522  reviewed)  studies  that  had 
at  least  one  contrast  between  DL  and  resident  learning.  Of  these  99  studies,  they 
were  able  to  calculate  50  independent  effect  sizes  (g).  Of  these  50  contrasts, 
they  arrived  at  a  mean  effect  size  of  +0.20  (p  <  0.0001)  in  favor  of  DL.  Therefore, 
DL  “produces  stronger  student  learning  outcomes  than  learning  solely  through 
face-to-face  [resident]  instruction”  (2013). 

C.  NPS  ACADEMIC  PROFILE  CODE  RESEARCH 

A  personal  interview  with  Ms.  Susan  Dooley,  NPS  Director  of  Admissions, 
revealed  that  her  awareness  of  the  Academic  Profile  Code  dates  back  to  no 
earlier  than  1983.  A  literature  review  of  theses  at  NPS  reveals  very  little 
information  on  the  origin  of  the  APC  as  an  admissions  tool  and  predictor  of 
student  success  at  NPS.  What  can  be  found  in  regards  to  the  APC  is  rather 
interesting.  The  earliest  study  to  involve  validation  of  the  APC  was  published  in 
1985. 
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1. 


Blatt 


Blatt  (1985)  is  the  first  published  thesis  in  Calhoun  (the  NPS  Institutional 
Digital  Archive)  to  discuss  and  analyze  the  APC.  Blatt  analyzed  each  component 
of  the  APC  (digit  1,  2,  and  3)  and  student’s  biographical  data  against  the  fourth 
quarter  QPR  of  159  Operations  Analysis  (OA)  students  at  NPS.  He  used  an 
Analysis  of  Variance  (ANOVA)  of  a  multiple  linear  regression  and  found  that 
APC1  (the  first  digit),  time  between  undergraduate  education  and  commencing 
NPS  studies,  Navy  officer  community,  and  a  student’s  undergraduate  degree 
were  determined  to  be  the  most  important  predictors  of  success  in  the  OA 
program  at  NPS. 

2.  Graduate  Record  Examination  Comparison 

As  a  side  note,  Soetrisno  (1975)  also  conducted  an  analysis  of  the 
performance  of  NPS  OA  students  and  used  biographical  data,  the  Strong 
Vocational  Interest  Blank  (SVIB;  a  psychological  test  used  in  career 
assessment).  Graduate  Record  Examination  (GRE)  scores,  and  the  Education 
Potential  Code  (EPC)  to  “develop  an  equation  predicting  academic  performance 
of  U.S.  Navy  officer  students.”  The  EPC  appears  to  have  been  the  predecessor 
of  the  APC;  therefore  at  some  point  between  1975  and  1983  the  EPC  was 
discarded  for  the  APC  as  the  primary  admissions  tool. 

Even  with  the  APC  as  the  primary  admissions  tool,  its  status  was  not  set 
in  stone.  Per  direction  from  the  Chief  of  Naval  Information  (CHINFO)  in 
September  1985,  all  students  entering  NPS  from  April  1986  to  April  1989  were 
required  to  take  the  GRE.  The  purpose  of  this  was  to  determine  if  the  GRE  score 
was  better  than  the  APC  as  an  indicator  of  intellectual  capability  and  predictor  of 
success  in  Master’s  programs  at  NPS  (Neil,  1989).  Three  theses  were  produced 
in  response  to  this  direction. 
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a.  Barr  and  Howard 

Barr  and  Howard  (1987)  were  the  first  NPS  students  to  produce  a 
response  to  CHINFO’s  request.  They  conducted  a  multivariate  analysis  of  age, 
sex,  years  since  receiving  bachelor’s  degree,  and  ARC  components  against  the 
Graduate  Quality  Point  Rating  (GQPR)  and  Total  Quality  Point  Rating  (TQPR)  of 
320  NPS  students  who  had  completed  their  third  quarter.  They  discovered  that  a 
student’s  age,  and  GRE  Verbal  and  Quantitative  scores  coupled  with  the  ARC 
first  digit  is  the  best  predictor  of  student  success.  Interestingly,  the  ARC  second 
and  third  digits  were  shown  not  to  be  useful  predictors  of  academic  success. 

b.  Transki 

Continuing  CHINFQ’s  study  on  the  possible  use  of  the  GRE  as  an 
entrance  requirement  for  NPS,  Transki  (1988)  conducted  a  multivariate  study 
similar  to  that  of  Barr  and  Howard  on  198  NPS  students  who  have  completed 
their  sixth  quarter.  She  concluded  that  the  GRE  is  a  much  stronger  predictor  of 
academic  performance  and  her  findings  were  in  agreement  with  the  previous 
GRE  study:  age,  GRE  scores  (including  Analytical),  and  the  ARC  first  digit  were 
the  best  predictors  of  student  success.  She  produced  a  formula  that  a  selection 
board  could  use  to  determine  an  applicant’s  prospective  success  at  NPS  with  an 
R^  of  0.361  and  a  standard  error  of  0.225  TQPR  points. 

c.  Neil 

Wrapping  up  the  three  year  study  on  the  GRE’s  utility  as  a  predictor  of 
student  performance  at  NPS,  Neil  (1989)  analyzed  the  performance  of  197  NPS 
students  and  validated  the  work  of  Barr,  Howard  and  Transki,  all  but  confirming 
the  GRE  alone  as  a  more  effective  measure  of  prospective  student  success,  but 
even  more  effective  if  combined  with  the  ARC.  She  produced  an  adjusted 
equation  to  better  determine  an  applicant’s  prospective  success  (that  includes  all 
ARC  digits  and  three  of  the  GRE  categories)  at  NPS  that  has  an  R^  of  0.414  and 
a  standard  error  of  0.285  TQPR  points. 
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3.  APC  Summary 

Despite  the  three-year  study  and  three  empirical  theses  demonstrating  the 
GRE  as  a  strong  supplement  to  prediction  of  student  success,  the  GRE  is  not 
currently  required  for  admittance  to  NPS.  There  appears  to  be  no  existing 
documentation  of  the  decision  to  forego  the  use  of  the  GRE  and  maintaining  the 
APC  as  the  primary  metric  for  entrance  into  programs  at  NPS. 

D.  LITERATURE  REVIEW  SUMMARY 

The  literature  review  displays  a  stark  contrast  in  the  results  of  civilian  DL 
to  resident  education  studies  when  compared  to  Navy  DL  versus  resident 
education  studies.  Civilian  studies  and  meta-analyses  hold  that  there  is  no 
measurable  difference  in  the  success  of  students  in  DL  when  related  to  resident 
student  success.  The  Navy  studies  indicate  that  DL  students  significantly  perform 
worse  than  resident  students  in  TA  and  NCPACE  education  programs  when 
reviewed. 

The  literature  reviewed  for  distance  learning  attrition  and  effectiveness 
focused  on  undergraduate  education  data  while  the  APC  reports  focused  on 
graduate  education.  In  addition  to  examining  the  APC,  this  study  also  focuses  on 
the  attrition  and  effectiveness  of  DL  and  resident  graduate  level  education  only.  It 
is  possible  that,  due  to  the  caliber  of  student,  there  is  an  unmeasurable  disparity 
between  the  academic  performance  of  undergraduate  and  graduate  level 
students. 
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III.  DATA 


The  data  used  to  conduct  the  analysis  in  this  study  was  provided  by  the 
NPS  Institutional  Research,  Reporting  and  Analysis  office  (IRRA).  The  data  was 
received  in  two  master  files,  Resident  student  data  and  DL  student  data.  To 
maintain  personal  security  measures  for  each  student  observed,  social  security 
numbers  were  not  part  of  the  data;  however  each  student’s  Identification  Number 
was  in  the  data.  Neither  we,  nor  anyone  outside  of  the  NPS  Institutional 
Research,  Reporting  and  Analysis  (IRRA)  office,  were  capable  of  using  Student 
Identification  Numbers  to  identify  individuals.  Data  was  further  safeguarded  as 
specified  in  the  approved  NPS  Institutional  Review  Board  (IRB)  protocol. 

The  Resident  file  contained  data  on  every  student  who  attended  NPS 
between  2008  and  2014.  The  DL  file  contained  data  on  every  student  who 
enrolled  in  a  DL  program  through  NPS  between  2006  and  2014.  There  were 
8,323  observations  comprising  all  foreign  and  U.S.  military  students  and  civilian 
students.  Each  student  observation  contained  67  variables,  including: 

•  U.S.  citizenship 

•  military  service 

•  military  pay  grade 

•  designator/Military  Occupation  Specialty  (MOS) 

•  undergraduate  graduation  year 

•  undergraduate  degree  name 

•  undergraduate  degree  granting  institution 

•  NPS  graduation  date 

•  Quality  Point  Rating  (Graduate,  Curriculum  and  Technical:  GQPR, 
CQPR,  TQPR) 

•  required  Academic  Profile  Code  (APC)  for  each  respective 
curriculum 

•  student’s  APC  (Naval  Postgraduate  School,  2014a) 

•  year  started  at  NPS 

•  enrollment  status 

•  if  graduated  “With  Distinction” 

Considering  the  fact  that  the  request  for  this  study  is  specifically  for  Navy 
officers  only,  we  reduced  the  data  to  Active  and  Reserve  Component  Navy 


19 


officer  students  at  NPS.  In  addition,  The  School  of  International  Graduate  Studies 
(SIGS)  does  not  offer  DL  programs;  therefore  all  students  from  that  school  have 
been  removed  from  the  analysis.  There  are  also  no  ‘Provost  Oversight’  (PO)  DL 
programs,  so  the  performance  of  those  74  students  was  not  part  of  the  analysis. 
This  resulted  in  a  total  of  2,633  observations  analyzed.  The  two  master  files 
(Resident  and  DL)  were  concatenated  into  one  master  file  with  an  additional 
column  added:  Distance  Learning  status. 

A.  PREDICTOR  VARIABLES 

Of  the  68  original  variables  provided  with  the  data,  only  26  were 
determined  to  be  probable  predictors  of  student  success.  Many  of  these  26 
columns  were  highly  correlated  with  one  or  more  other  columns. 

1.  Distance  Learning  Status 

This  is  the  first  of  two  major  predictors  of  interest  in  this  study.  It  is  a 
simple  binary  column  detailing  if  students  were  enrolled  in  a  distance  learning 
program  or  attended  courses  on  campus  at  NPS.  Of  all  the  Navy  officers  to  take 
courses  from  NPS  in  the  provided  time  windows,  1,187  were  DL  and  1,787  were 
Resident. 


Figure  1 .  Breakdown  of  distance  learning  and  resident  NPS  students 

(after  IRRA,  2014). 
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2.  Military  Pay  Grade 

The  spread  of  ranks  among  students  taking  courses  from  NPS  during  the 
time  frame  ranged  from  E-6  to  0-7.  Even  though  the  request  for  this  study 
specifically  stated  Navy  officers,  we  decided  to  retain  the  handful  of  enlisted 
personnel  (6  total;  2  E-6  and  4  E-7)  to  gain  insight  on  their  success  at  NPS. 


Figure  2.  Breakdown  of  NPS  students’  Navy  pay  grade.  The  majority  of 
students  are  of  rank  0-3  (after  IRRA,  2014). 

3.  Designator/Community 

The  original  column  for  this  predictor  listed  the  numeric  designator  (i.e. 

1110  for  Surface  Warfare  Officer  or  1310  for  Pilot)  for  each  observation;  for 

enlisted  personnel,  the  Navy  Enlisted  Classification  (NEC)  code.  This  designator 

column  contains  90  separate  numeric  levels,  a  number  we  determined  to  be  too 

cumbersome  and  unlikely  to  provide  useful  insight.  With  this  in  mind,  we 

developed  a  new  column,  “Community,”  and  translated  each  designator  to  its 

broader  Navy  officer  community,  which  resulted  in  11  levels  (the  NEC  of  the 

enlisted  personnel  were  all  computer  network  administration/defense  skills  so  we 

placed  all  the  enlisted  personnel  into  the  Information  Dominance  Corps  [IDC]). 

The  “Community”  predictor  can  provide  insight  into  which  Navy  officer 

communities  have  a  higher  probability  of  success  at  NPS. 
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Figure  3.  Breakdown  of  NFS  students’  Navy  officer  Communities.  The 
majority  of  students  are  from  the  aviation  field  (after  IRRA,  2014). 

4.  Undergraduate  Degree  Year  and  Start  Academic  Year 

These  two  separate  predictors  provide  no  insight  while  standing  alone,  but 
the  difference  between  the  two  columns  provides  the  number  of  years  between 
the  time  each  student  earned  his  or  her  bachelor’s  degree  and  when  he  or  she 
started  working  toward  a  master’s  degree  with  NPS.  Therefore,  we  produced  a 
new  predictor,  “SinceUGrad,”  that  lists  the  time,  in  years,  between  undergraduate 
completion  and  starting  work  on  a  master’s  degree  with  NPS.  It  is  possible  that 
this  predictor  can  have  negative  correlation  with  student  performance.  We  also 
kept  “StartAcadYear”  in  the  analysis  to  see  if  any  variability  in  student  success  is 
dependent  on  the  year  the  student  began  his  or  her  studies  at  NPS. 
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Figure  4.  Distribution  of  years  between  undergraduate  degree  and 
starting  school  at  NFS.  Most  students  have  been  out  of  school 
for  more  than  5  years  (after  IRRA,  2014). 


5.  Undergraduate  School  Name 

This  column  listed  each  of  the  511  separate  institutions  from  which 
students  earned  bachelor’s  degrees.  We  determined  511  levels  to  be  too 
cumbersome  for  insightful  analysis  so  we  produced  a  new  column  with  a  binary 
response:  whether  or  not  the  student  graduated  from  the  United  States  Naval 
Academy  (USNA).  This  column,  “USNAgrad,”  can  provide  insight  into  whether  or 
not  USNA  graduates  have  a  greater  probability  of  success  at  NPS. 
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Figure  5.  Breakdown  of  Naval  Academy  graduates  compared  to  other 
secondary  institutions  of  Navy  officer  NFS  students  (after  IRRA, 

2014). 


6.  NFS  School  Name 

Given  that  there  are  66  distinct  NPS  curricula  listed  in  this  data,  we 
decided  to  use  the  curricula’s  parent  school  at  NPS.  This  can  provide  insight  into 
which  of  the  three  NPS  schools  with  DL  programs  have  more  successful 
students. 


Figure  6.  Breakdown  of  Navy  officer  students  by  NPS  school  attended. 
The  majority  of  students  study  in  the  GSBPP  (after  IRRA,  2014). 
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7.  Required  ARC  and  Student’s  ARC 

This  is  the  second  of  two  major  predictors  of  interest  in  this  study.  Given 
that  the  ARC  is  a  determination  of  a  student’s  performance  in  his  or  her 
respective  undergraduate  program  and  is  compared  to  a  baseline  ARC  for  each 
curriculum,  it  is  reasonable  to  assume  that  the  further  from  the  baseline  ARC  a 
student’s  ARC  is  from  his  or  her  program’s  required  ARC,  the  better  or  worse 
he/she  can  be  expected  to  perform  at  NRS.  With  this  in  consideration,  we 
produced  another  column,  “ARCdelta,”  that  lists  the  sum  of  the  differences 
between  each  digit  of  the  student’s  ARC  and  required  ARC.  For  example,  if  a 
curriculum  has  a  required  ARC  of  222,  and  an  applicant  has  an  ARC  of  112,  her 
ARC  delta  is  +2.  If  another  applicant  has  an  ARC  of  132,  his  ARC  delta  is  0.  In 
this  second  case,  the  -1  on  the  first  digit  and  the  +1  on  the  second  digit  do  cancel 
out.  This  indicates  that,  even  though  the  applicant’s  ARC  is  inadequate  for  one  of 
the  ARC  digits,  the  other  better  than  required  ARC  digit  makes  up  for  that  deficit. 
We  recognize  that  applicant  may  have  required  a  waiver  but,  holistically,  met  the 
requirement  for  that  program. 

The  higher  the  number  in  “ARCdelta,”  the  better  the  student  did  as  an 
undergraduate  student  compared  to  the  ARC  required  by  his  or  her  program.  The 
lower  number  in  “ARCdelta,”  the  worse  the  student  did  as  an  undergrad 
compared  to  the  ARC  required  by  his  or  her  program.  This  is  expected  to 
produce  a  positive  correlation  between  the  ARC  delta  and  student  success  and 
provide  insight  as  to  whether  the  ARC  is  a  valid  predictor  of  student  success  at 
NRS. 
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Figure  7.  Distribution  of  APCdeltas  among  NFS  Navy  officer  students. 
Notice  the  slightly  right-skewed,  bell  shape  (after  IRRA,  2014). 


8.  Refresher  Quarter 

This  column  is  a  binary  indicator  of  whether  or  not  a  student  started  his  or 
her  time  at  NPS  with  a  refresher  quarter. 

This  [refresher  quarter]  is  a  sequence  of  courses  developed  by  the 
Program  Officer  and  the  Academic  Associate  to  better  prepare 
incoming  students  for  entering  a  technical  curriculum.  This  course 
sequence  is  designed  for  prospective  students  who  have  an 
Academic  Profile  Code  (ARC)  that  indicates  a  deficiency  in 
mathematics  and/or  scientific  and  technical  subject  matter  (i.e., 
their  ARC  does  not  qualify  them  for  direct  entry  to  a  technical 
curriculum)  or,  in  completing  their  review  of  the  prospective 
student’s  academic  record,  the  Program  Officer  and  Academic 
Associate  have  concluded  that  sufficient  time  has  expired  since  the 
student’s  most  recent  college  experience  and  as  such,  the  student 
would  benefit  from  the  Technical  Refresher  Quarter.  For  some 
students,  this  may  also  include  courses  from  the  Six-Week  Math 
Refresher.  The  refresher  sequence  is  normally  twelve  weeks  in 
length;  however,  there  are  occasions  when  a  student  may  be 
assigned  two  quarters  of  refresher  prior  to  entering  a  technical 
curriculum.  (Naval  Postgraduate  School,  2014) 

The  “Refresher”  predictor  is  expected  to  have  positive  influence  on 
academic  success  at  NPS. 
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Figure  8.  Breakdown  of  NFS  Navy  officer  students  who  took  one  or 
more  “refresher  quarters”  before  starting  official  studies  (after 

IRRA,  2014). 

9.  Retake  Hours  and  Graduate  Retake  Hours 

Two  separate  columns  in  the  original  data  list  how  many  hours  each 
student  had  to  retake  while  attending  NPS.  Rather  than  analyzing  the  number  of 
retake  hours  a  student  achieved,  we  decided  to  create  a  new  column,  “Retake,” 
that  provided  a  binary  response  on  whether  or  not  the  student  had  to  retake  any 
courses.  This  is  expected  to  have  a  negative  correlation  with  student  success. 


Retake 
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Figure  9.  Breakdown  of  NPS  Navy  officer  students  who  had  to  repeat  a 

class  (after  IRRA,  2014). 
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B.  RESPONSE  VARIABLES 


The  biggest  challenge  of  this  study  is  selecting  the  response  variable  that 
best  represents  academic  success.  Of  the  68  columns  in  the  original  data,  I 
determined  7  of  them  to  be  possible  definers  of  academic  success. 

1.  Quality  Point  Rating 

Generally  referred  to  as  the  GPA,  the  QPR  at  NPS  is  calculated  by  taking 
the  sum  of  the  quality  points  for  all  courses  and  dividing  it  by  the  sum  of  the 
quarter-hour  credits  for  those  courses.  This  gives  a  weighted  numerical 
evaluation  of  the  student’s  performance.  (Naval  Postgraduate  School,  2014). 
NPS  tracks  three  separate  types  of  QPR. 

a.  Curriculum  Quality  Point  Rating 

The  CQPR  is  the  QPR  calculated  for  grades  received  by  a  student  for 
courses  taken  as  part  of  his  or  her  designated  curriculum. 

b.  Graduate  Quality  Point  Rating 

The  GQPR  is  the  QPR  calculated  for  grades  received  by  a  student  in  his 
or  her  3000  and  4000  level  courses.  Students  must  have  a  GQPR  above  3.00  to 
be  eligible  to  earn  a  degree  and  graduate  from  NPS.  We  created  this  column  by 
setting  all  GQPRs  below  3.00  to  N/A  so  that  we  can  analyze  the  population  of 
Resident  and  DL  students  that  satisfactorily  complete  their  required  course  of 
study  and  if  there  is  any  difference  between  the  two  groups  of  NPS  graduates. 

c.  Total  Quality  Point  Rating 

The  TQPR  is  the  QPR  calculated  for  grades  received  by  a  student  in  all 
courses  taken  through  NPS.  We  decided  to  use  the  TQPR  as  the  basis  for 
defining  student  success  with  QPR  because  it  shows  the  overall  performance  of 
a  student  with  NPS. 
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Figure  10.  Distribution  of  NFS  Navy  officer  student  TQPR  by  resident  and 
DL  students.  There  is  a  significant  number  of  TQPRs  in  the  0.0  to 
0.5  range  on  the  DL  chart  (after  IRRA,  2014). 


2.  Graduation  Eligible 

This  is  a  binary  response  listing  whether  or  not  a  student  has  a  GQPR 
above  3.0.  As  per  the  NPS  Student  Handbook  (2014),  a  student  must  have  a 
GQPR  above  3.0  and  a  TQPR  above  2.75  to  be  eligible  to  graduate  from  NPS. 
We  decided  that  GQPR  (based  on  grades  in  3000+  level  classes)  is  the  more 
critical  of  the  two. 
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Figure  1 1 .  Distribution  of  NFS  Navy  officer  student  GQPRs  with  the 
“Graduation  Eligible”  and  “Not  Graduation  Eligible”  line  at  the  3.0 
mark  (after  IRRA,  2014). 


3.  “With  Distinction” 

This  is  a  binary  response  listing  whether  or  not  a  student  received  his  or 
her  degree  “with  distinction”  at  the  time  of  graduation.  As  per  the  NPS  Academic 
Policy  manual: 

The  Academic  Council  will  recommend  certain  students  receiving 
master’s  degrees  to  the  President  for  the  award  of  their  degrees 
With  Distinction.  The  students  must  be  nominated  to  the  Academic 
Council  by  the  cognizant  academic  unit.  Academic  units  are 
encouraged  to  develop  criteria  beyond  the  Quality  Point  Rating  to 
evaluate  outstanding  student  performance.  To  be  eligible  for  a 
master’s  degree  With  Distinction,  the  student  must  have  earned  a 
minimum  of  24  quarter-hours  of  graduate  level  courses  presented 
for  his  or  her  degree.  In  any  one  academic  year  no  more  than  ten 
percent  (or  one  student,  whichever  is  larger)  of  the  students 
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earning  a  master’s  degree  in  the  degree  programs  of  the 
nominating  academic  unit  shall  be  nominated  for  degrees  “with 
distinction.”  (Naval  Postgraduate  School,  2014b) 

No  more  than  10%  of  graduating  students,  by  school,  may  selected  for 
graduation  with  distinction  (Naval  Postgraduate  School,  201 4d). 


Figure  12.  Breakdown  of  Resident  Navy  officer  students  who  graduated 

“with  distinction”  (after  IRRA,  2014). 


Figure  13.  Breakdown  of  DL  Navy  officer  students  who  graduated  “with 

distinction”  (after  IRRA,  2014). 
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4. 


Enrollment  Status/Date 


These  two  possible  responses  are  on  the  other  end  of  the  student  success 
spectrum.  The  “EnrollStatus”  column  lists  each  student’s  positive  status 
(Standard)  or  negative  enrollment  status  (Disenrolled)  either  for  academic, 
punitive,  or  administrative  non-punitive  reasons.  The  “DisEnrollDate”  column  lists 
the  date  disenrolled  students  were  released  from  studies  with  NPS.  I  decided  to 
consider  any  disenrollment  from  NPS  as  student  failure  and  therefore  produced 
the  column  “disenrolled”  as  a  binary  response  listing  whether  or  not  a  student 
was  disenrolled.  This  differs  from  “Graduation  Eligible”  because  some  with 
GQPR  scores  less  than  3.0  were  still  in  school  at  the  time  of  data  collection  and 
still  had  the  opportunity  to  raise  their  GQPR  into  the  graduation-eligible  range. 


Figure  14.  Breakdown  of  resident  Navy  officers  students  who  were 
disenrolled  from  NPS  (after  IRRA,  2014). 
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Figure  15.  Breakdown  of  DL  Navy  officers  students  who  were  disenrolled 

from  NFS  (after  IRRA,  2014). 
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IV.  ANALYSIS 


As  previously  stated,  this  study  has  a  dual  purpose.  The  first,  and 
overarching,  endeavor  of  this  study  is  to  determine,  through  empirical  analysis  of 
the  available  data,  whether  or  not  students  from  the  same  population  pool  (  Navy 
officers)  reach  equal  levels  of  success  in  Distance  Learning  versus  Resident 
education  programs  at  NPS.  The  purpose  of  the  regression  models  developed  by 
this  study  is  not  to  provide  predictive  capability  but  to  ascertain  which  predictors 
significantly  contribute  to  student  success  or  failure.  In  fact,  we  aim  to  find 
evidence  to  address  the  null  hypothesis  that  DL  student  performance  is  equal  to 
Resident  student  performance. 

The  second  purpose  of  this  study  is  to  use  the  same  regression  models  to 
maintain  the  establishment  of  the  NPS  Academic  Profile  Code  as  a  valid 
predictor  of  student  success  by  analyzing  the  students’  own  APC  and  its 
composite  deviance  from  their  respective  programs’  required  APC. 

“Academic  success”  is  an  entirely  subjective  term;  therefore,  it  is 
intellectually  insincere  to  select  one  single  metric  to  define  this  abstract.  To  many 
students  working  toward  a  Master’s  Degree  through  NPS,  the  accomplishment  of 
graduation  (having  a  GQPR  greater  than  or  equal  to  3.00)  is  considered  a 
success.  Others  may  believe  their  academic  success  must  be  validated  with  an 
award  such  as  graduating  “With  Distinction.”  Therefore,  by  using  the  R  statistical 
computing  program  (R  Core  Team,  2014),  we  developed  models  to  analyze  the 
selected  predictors  with  TQPR  and  “With  Distinction”  as  the  response.  In 
addition,  we  produced  regression  and  classification  trees  through  Recursive 
Partitioning  with  all  TQPR  data  and  two  subsets  of  the  TQPR  data  (graduate 
eligible  students  and  disenrolled  students)  as  the  response.  The  predictors  within 
this  model  (and  all  models  developed  in  this  study)  are  listed  in  Table  4. 
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Table  4.  Predictor  variables  used  in  all  models  (after  IRRA,  2014). 


Predictor  Variable 

Baseline  Category 

Categories  within  Model 

Distance  Learning 

No  (Resident) 

Yes 

Pay  Grade  (factor) 

0-3 

E-6,  E-7,  0-1,  0-2,  0^,  0-5,  0-6, 
0-7 

Community  (factor) 

Surface  Warfare 
(SWO) 

Aviation,  Submarine,  Special 
Operations,  Human  Resources, 
Supply,  Engineer,  Information 
Dominance,  Medical, 
Miscellaneous,  Limited  Duty  Officer 

Start  Academic  Year 

N/A 

2006  to  2012 

Since  Undergrad 

N/A 

1  to  38  years 

USNA  Graduate 

No 

Yes 

NPS  School  Name 

GSBPP 

GSEAS,  GSEAS 

APC  Delta 

N/A 

minus  11  to  12 

Class  Retake 

No 

Yes 

Refresher  Quarter 

No 

Yes 

A.  TOTAL  POPULATION 

The  first  look  at  how  any  of  these  ten  predictors  may  affect  student 
success  is  to  develop  a  simple  linear  model  (LM)  with  every  Navy  officer  student 
in  NPS  colleges  that  offer  resident  and  DL  courses,  a  total  of  2,633  observations. 

1.  TQPR  as  Response  Variable 

An  NPS  student’s  TQPR  is  a  strong  indicator  of  his  or  her  performance  in 
all  classes  taken  through  NPS.  It  directly  correlates  to  a  student’s  final  grade  in  a 
given  class.  For  example,  a  grade  of  A  earns  four  points,  a  grade  of  B  earns  3 
points,  while  in  between  those  levels,  a  grade  of  A-minus  earns  3.7  points,  and  a 
B-plus  earns  3.3  points.  This  follows  suit  throughout  the  rest  of  the  grade 
spectrum  down  to  a  grade  of  D  receiving  1  point.  The  TQPR  is  an  average  of 
those  points  earned  for  all  classes  taken  (Naval  Postgraduate  School,  2014).  A 
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simple  linear  regression  model  is  developed  v^ith  TQPR  (a  continuous  response 
variable  between  0  and  4)  as  the  response.  The  results  of  this  model  are  in  Table 
5.  It  provides  the  estimate  of  the  coefficients,  the  standard  error,  t-value,  and  p- 
value  (Pr(>|t|))  for  each  predictor  variable.  All  information  in  the  following  tables 
are  print-outs  from  the  R  program. 


Table  5.  Summary  results  for  the  simple  linear  regression  of  the  total 
population.  TQPR  points  as  response  variable.  Eleven 
variables  have  strong  significance  (after  IRRA,  2014). 


Coefficients : 

Estimate 

Std.  Error 

t-value 

PrOltl) 

Intercept 

-5.598 

14.445 

-0.388 

0.698 

DL 

-0.323 

0.036 

-8.971 

<2e-16 

it  ★  ★ 

E-6 

0.013 

0.418 

0.031 

0.975 

E-7 

0.182 

0.347 

0.524 

0.600 

0-1 

-0.808 

0.160 

-5.056 

4.64e-07 

★  ★ 

0-2 

-0.122 

0.105 

-1.164 

0.244 

0-4 

0.103 

0.039 

2.626 

0.009 

*  •* 

0-5 

0.120 

0.065 

3.083 

0.002 

** 

0-6 

0.828 

0.145 

5.713 

1.26e-08 

*  *  * 

0-7 

0.800 

0.604 

1.324 

0.186 

Aviator 

0.427 

0.042 

10.134 

<2e-16 

*  ★  * 

Nuke 

0.023 

0.058 

0.397 

0.691 

SOF 

0.260 

0.102 

2.534 

0.011 

★ 

HR 

0.155 

0.076 

2.117 

0.034 

* 

Supply 

0.152 

0.059 

2.578 

0.100 

* 

Engineer 

0.209 

0.050 

4.213 

2.62e-05 

*  *  * 

IDC 

0.272 

0.049 

5.586 

2.60e-08 

★  ★  * 

Med 

0.243 

0.080 

3.026 

0.003 

*  ★ 

Misc 

0.202 

0.114 

1.771 

0.770 

, 

LDO 

-0.107 

0.185 

-0.580 

0.562 

start  Acad  Year 

0.005 

0.007 

0.643 

0.520 

Since  Ugrad 

-0.024 

0.005 

-4.908 

9.89e-07 

^r■k* 

USNA  Grad 

0.016 

0.029 

0.546 

0.585 

GSEAS 

-0.206 

0.037 

-5.526 

3.66e-07 

*★* 

GSOIS 

-0.245 

0.005 

-6.674 

3.13e-ll 

*  ★  ★ 

APC  delta 

0.033 

0.041 

7.043 

2.50e-12 

*  ★  * 

Refresher 

0.064 

0.041 

1.547 

0.122 

Retake 

-0.352 

0.101 

-3.486 

0.001 

★  ★  * 

Signif  Codes: 

0  ■***’  0.001 

'**■  0.01 

0.05  ’. 

■  0.1  '  ' 

Residual  SE: 

0 

.5875  on  2227 

df 

Mult  R''2: 

0 

.1673 

Adj  R''2: 

0.1572 

F-stat : 

16.58  on  27  &  2227  df 

p-value : 

<2.2e-16 
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This  LM  model  indicates  that,  all  other  variables  being  equal,  a  DL  student 
will  have  a  TQPR  that  is  0.32  points  less,  on  average,  than  a  resident  student. 
The  other  variable  of  primary  interest  shows  that  a  1  point  increase  in  APC  delta 
is  associated  with  an  average  0.03  point  increase  in  TQPR  (again,  all  other 
variables  being  equal).  In  addition  to  the  strong  significance  of  DL  and  APC  delta, 
the  other  variables  of  interest  that  show  strong  significance  are  the  military  pay 
grades  0-1  and  0-6,  the  Aviation,  Engineer,  and  IDC  communities,  years  since 
undergraduate  degree,  graduate  school  attended,  and  retaking  a  class.  To 
ensure  thorough  analysis,  we  decided  to  run  an  Analysis  of  Variance  (ANOVA)  to 
see  how  strong  each  predictor  is  without  every  other  factor  listed.  The  results  are 
in  Table  6. 


Table  6.  ANOVA  table  for  total  population.  TQPR  LM.  Seven 
predictors  have  strong  significance  (after  IRRA,  2014). 


Deg  Frdm 

Sum  Sq 

Mean  Sq 

F  value 

Pr  OF) 

DL 

1 

12.75 

12.75 

36.94 

1.43e-09 

★  *  * 

Pay  Grade 

8 

26.61 

3.33 

9.64 

3.28e-13 

★  ★ 

Community 

10 

60.05 

6.01 

17.40 

<2.2e-16 

*  *  ★ 

Start  Acad  Year 

1 

0.12 

0.12 

0.36 

0.550 

Since  Ugrad 

1 

11.05 

11.05 

32.01 

1.73e-08 

★  *  * 

USNA  Graduate 

1 

0.38 

0.38 

1.11 

0.290 

School 

2 

21.66 

10.83 

31.37 

3.65e-14 

**  * 

APC  delta 

1 

16.86 

13.86 

48.84 

3.66e-12 

*  *  * 

Refresher 

1 

0.80 

0.80 

2.31 

0.128 

Retake 

1 

4.19 

4.19 

12.15 

5.00e-04 

*  *  ★ 

Residuals 

2227 

768.65 

0.35 

Signif  Codes : 

0  •***'  0.001 

'**'  0.01 

0.05  '  .  ' 

0.1  '  ' 
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This  ANOVA  provides  evidence  that  all  but  three  predictors  (Start 
Academic  Year,  USNA  Graduate,  and  Refresher  Course  taken)  are  very 
significant  determinants  of  a  student’s  TQPR. 

The  identification  of  multiple  variables  as  strongly  significant  and  the  high 
F-statistic  (16.58  on  27  and  2,227  degrees  of  freedom)  indicates  that  this  model 
is  better  than  the  null,  intercept  only  model.  Low  multiple  and  adjusted  R-squared 
(0.17  and  0.16,  respectively)  values  lend  credence  to  the  assumption  that  there 
is  a  lot  of  noise  in  the  data.  More  exploration  is  essential  to  uncover  any  real 
insight  into  the  data. 

2.  Recursive  Partitioning  (TQPR  as  Response  Variabie) 

Recursive  partitioning  is  a  technique  that  is  used  to  determine  where 
interactions  exist  within  the  data.  This  method  uses  a  two  stage  process  to  build 
a  classification  or  regression  model  that  can  be  represented  as  a  binary  tree.  The 
first  stage  determines  which  single  variable  best  splits  the  data  into  two  groups 
by  maximizing  the  reduction  in  Gini  impurity,  “a  measure  of  how  often  a  randomly 
chosen  element  from  the  set  would  be  incorrectly  labeled  if  it  were  randomly 
labeled  according  to  the  distribution  of  labels  in  the  subset”  (Strickland,  2014). 
Once  the  data  is  split,  this  same  process  determines  the  next  level  of  best  split 
for  each  successive  group  of  variables.  This  can  continue  until  no  improvement 
can  be  made.  To  avoid  developing  an  overly  complex  model  (which  provides  no 
insight),  the  second  stage  uses  cross-validation  to  “trim  back  the  full  tree” 
(Therneau  &  Atkinson,  2014). 

The  first  stage  in  recursive  partitioning  can  be  conducted  by  the  computer 
while  the  second  stage  requires  the  analyst  to  determine  where  the  cross 
validated  error  is  the  lowest  and  provides  a  regression  tree  (RT)  that  clearly 
shows  where  the  important  interactions  occur.  The  results  of  this  process  for  the 
TQPR  response  variable  are  in  Figure  16.  It  provides  the  hierarchical  position  of 
each  node  and  its  successive  child  nodes,  which  variable  is  split  and  where,  the 
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number  of  observations  in  that  node,  the  deviance  of  the  observations  within  that 
node,  and  the  average  value  of  the  respondent  which  in  this  case  the  TQPR. 


YES 

Submariner  or  LDO? 

N=2633  TQPR=3.54  _ 

NO 

Distance  Lear 
N=293 

ning  Student?  N=2340 

TQPR=3.02  TQPR=3.60 

GSEAS  0 
N=170 

rGSOIS?  N=123 

TQPR=2.51  TQPR=3.74 

N=140  N=30 

TQPR  =  2.24  TQPR=3.73 

Figure  16.  RP  regression  tree  of  the  total  population.  TQPR  as  response 
variable.  Average  TQPR  of  140  DL  students  in  GSEAS  and 
GSQIS  is  2.24  (after  IRRA,  2014). 


Qne  can  see  in  Figure  16  that  the  first  single  variable  that  best  splits  the 
data  into  two  groups  is  Navy  officer  community,  where  Submariners  (“NUKE”) 
and  Limited  Duty  Qfficers  (“LDQ”)  are  identified  as  a  significant  group.  All  other 
Navy  officer  communities  are  grouped  together  and  this  group  ends  in  a  terminal 
node  with  an  average  TQPR  of  3.60.  The  “NUKE”  and  “LDQ”  subset  continues 
on  to  find  the  second  single  variable  that  best  splits  it.  This  time  it  is  into  DL 
students  (lsDL=Y)  and  resident  students  (lsDL=N,  which  ends  in  another  terminal 
node,  with  an  average  TQPR  of  3.74).  The  last  split  in  this  recursive  partition  is 
into  two  terminal  nodes.  Qne  node  has  GSBPP  alone  with  an  average  TQPR  of 
3.73.  The  other  terminal  node  is  GSQIS  and  GSEAS  with  an  average  TQPR  of 
2.24. 
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We  can  now  see  an  interaction  that  identifies  a  subset  of  the  total 
population  with  low  TQPR;  Limited  Duty  Officers  and  Submariners  in  a  DL 
program  through  GSEAS  and  GSOIS,  a  group  of  140  students,  have  an  average 
TQPR  of  2.24.  This  indicates  a  DL  group  that  does  not  have  academic  success. 

3.  Graduation  Eligible  as  Response  Variable 

Within  this  entire  population,  some  students  had  a  GQPR  below  3.0; 
therefore  they  were  not  eligible  to  graduate  at  the  time  of  the  data  collection.  To 
see  if  any  of  the  predictors  can  identify  students  that  are  not  able  to  graduate,  we 
set  the  binary  Graduation  Eligible  as  the  response:  yes  (GQPR  greater  than  or 
equal  to  3.0)  or  no  (GQPR  less  than  3.0).  In  addition,  the  small  number  of 
observations  of  pay  grades  E-6,  E-7,  and  Q-7  were  resulting  in  absurdly  large 
standard  errors  for  those  pay  grades.  Therefore,  we  suppressed  those  pay 
grades  in  the  model  to  allow  for  a  preferable  model. 

Given  the  fact  that  the  response  to  this  model  is  binary,  we  calculated  the 
marginal  effects  (the  predicted  probability  associated  with  a  one-unit  increase  in 
a  predictor  when  all  other  variables  are  set  to  their  mean  values)  to  determine 
how  much  influence  each  variable  has  on  the  result.  This  was  done  by  using  the 
R  package  “mfx”  (Fernihough,  2015).  The  results  of  this  model  are  shown  below 
in  Table  7.  It  provides  the  estimate  of  the  coefficients,  the  marginal  effect,  the 
standard  error,  z-value,  and  p-value  (Pr(>|z|))  for  each  predictor  variable. 
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Table  7.  Summary  results  for  the  binomial  LM  of  the  total  population. 
“Graduation  Eligible”  as  response  variable.  Five  variables  have 
strong  significance  (after  IRRA,  2014). 


Coefficient's : 

Estimate 

Mrg.  Effect 

Std.  Error 

z-value 

PrOlzl) 

Intercept 

-14.765 

126.659 

-0.117 

0.907 

DL 

-2.743 

-0.057 

0.405 

-6.779 

1.21e-ll 

0-1 

-0.964 

-0.020 

0.724 

-1.333 

0.182 

1 

o 

0.663 

0.006 

0.718 

0.924 

0.356 

0-4 

0.549 

0.006 

0.401 

1.372 

0.170 

0-5 

1.007 

0.009 

0.559 

1.803 

0.716 

0-6 

2.674 

0.012 

1 . 107 

2.415 

0.016 

Aviator 

1.653 

0.018 

0.362 

4.564 

5.02e-06 

Nuke 

-0.504 

-0.008 

0.417 

-1.207 

0.227 

SOF 

0.774 

0.007 

1.190 

0.651 

0.515 

HR 

1.211 

0.006 

0.717 

1.689 

0.910 

Supply 

1.159 

0.010 

0.817 

1.420 

0.156 

Engineer 

0.632 

0.006 

0.507 

1.245 

0.213 

I  DC 

0.917 

0.008 

0.484 

1.896 

0.058 

Med 

2.013 

0.012 

0.815 

2.470 

0.014 

Mi  sc 

-0.319 

-0.005 

0.658 

-0.485 

0.628 

LDO 

-0.678 

-0.012 

1.019 

-0.666 

0.505 

Start  Acad  Year 

0.010 

0.000 

0.063 

0.163 

0.871 

since  Uqrad 

-0.119 

-0.015 

0.038 

-3.129 

0.002 

USNA  Grad 

0.045 

0.001 

0.285 

0.157 

0.871 

GSEAS 

-1.233 

-0.021 

0.380 

-3.246 

0.001 

GSOIS 

-2.112 

-0.052 

0.350 

-6.043 

1.51e-09 

ARC  delta 

0.149 

0.002 

0.044 

3.413 

0.001 

Refresher 

0.091 

0.001 

0.524 

0.173 

0.863 

Retake 

-2.264 

-0.095 

0.549 

-4.210 

3.79e-05 

S i gn i f  Code  s : 

0  »***•  0.001 

’**'  0.01 

0.05  * . ■  0.1 

' 

Null  Dev:  787.16  on  2248  df 
Residual  Dev:  548.37  on  2224  df 
AIC:  598.37 
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Many  fewer  predictors  are  showing  strong  significance  in  this  binomial 
response  logistic  model.  The  only  variables  that  carried  over  from  the  TQPR  LM 
and  continue  into  this  model  are  the  Aviation  community,  GSOIS,  retaking  a 
class,  DL  and  APC  delta.  Since  this  is  a  binomial  response,  we  analyze  the 
marginal  effects  of  each  variable.  In  this  case,  the  logistic  model  indicates  that 
DL  students  are  5.7%  less  likely  to  be  graduation  eligible;  one-point  increase  in 
APC  delta  improves  the  likelihood  of  graduation  eligibility  by  only  0.2  percent. 
The  results  of  an  Analysis  of  Variance  (ANOVA)  calculated  with  the  ‘MASS’ 
package  (Ripley,  2015)  in  R  (due  to  the  binomial  response)  for  this  model  are  in 
Table  8. 


Table  8.  ANOVA  Table  of  the  total  population.  “Graduation  Eligible” 
binomial  LM.  five  predictors  have  strong  significance  (after 

IRRA,  2014). 


Deg  Frdm 

Deviance 

AIC 

LRT 

Pr(Chi) 

NULL 

548. S'? 

604.37 

DL 

1 

606.83 

660.83 

58.46 

2.07e-14 

Pay  Grade 

8 

559.60 

599.60 

11.23 

0.189 

Community 

10 

594.20 

630.20 

45.83 

1.54e-06 

start  Acad  Year 

1 

548.40 

602.40 

0.03 

0.871 

Since  Ugrad 

1 

548.67 

612.67 

10.30 

0.001 

USNA  Graduate 

1 

548.40 

602.40 

0.03 

0.875 

School 

2 

588.12 

640.12 

39.75 

2.34e-09 

APC  delta 

1 

560.31 

614.31 

11.94 

0.001 

Refresher 

1 

548.40 

602.40 

0.03 

0.863 

Retake 

1 

561.37 

615.37 

13.00 

0.000 

Signif  Codes: 

0  '***'  0.001 

•**'  0.01 

0.05  '. 

0.1  '  ' 

This  ANOVA  table  is  very  similar  to  TQPR’s  ANOVA  table  where  the  same 
three  predictors  are  not  significant,  and  the  variable  “MilitaryPayGrade”  was 
reduced  from  high  significance  to  no  significance  at  all.  This  model  analyzed  the 
same  amount  of  data  as  the  TQPR  LM,  therefore  the  high  level  of  noise  still 
exists,  but  dividing  the  model  into  two  separate  groups  (GQPR  less  than  3.0  and 
GQPR  greater  than  or  equal  to  3.0)  has  reduced  the  number  of  variables  that 
show  strong  significance. 
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4.  Recursive  Partitioning  (Binomial  Response) 

In  addition  to  a  continuous  response  variable,  recursive  partitioning  is  also 
capable  of  developing  a  classification  tree  for  a  binomial  response.  The  response 
is  either  yes  or  no  in  all  binomial  response  models  produced  in  this  study.  The 
results  of  recursive  partitioning  with  “Graduation  Eligible”  (binomial)  are  in  Figure 
17.  It  provides  the  hierarchical  position  of  each  node  and  its  successive  child 
nodes,  which  variable  is  split,  the  number  of  observations  in  that  node,  and  the 
probability  of  that  respondent  occurring  for  a  “yes”  or  “no”  response  to  graduation 
eligibility. 


YES 

Submariner  or  LDO? 

N=2633  Prob=94.2% 

NO 

Distance  Lean 
N=293  1 

ling  Student?  N=2340 

Prob=79.5%  Prob=96.0% 

GSEAS  0 
N=170  I 

r  GSOIS?  N=123 

Prob=65.9%  Prob=98.4% 

0-3,  O- 
N=14(D 

4,  0-5?  N=30 

I  Prob=59.2%  TQPR=96.7% 

N=93  IN 

Prob=52.7%  Prob= 

1=47 

=83.0% 

Figure  17.  RP  classification  tree  of  the  total  population.  “Graduation 
Eligible”  as  response  variable.  Probability  of  being  graduation 
eligible  for  93  mid-grade  Navy  officers  in  DL  programs  in  GSEAS 
and  GSOIS  is  52.7%  (after  IRRA,  2014). 


This  RP  produces  a  classification  tree  that  goes  one  step  further  in 
complexity  than  the  TQPR  response  RP  by  identifying  the  military  pay  grades  of 
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0-3,  0-4,  and  0-5  as  a  group  of  factors  in  the  “MilitaryPayGrade”  variable  that 
have  a  reduced  probability  of  being  graduation  eligible.  This  CT  indicates  that 
Submarine  and  LDO  Navy  officers  of  the  ranks  0-3  through  0-5  taking  DL 
courses  in  either  the  GSOIS  or  GSEAS  have  a  52.7%  probability  of  being  eligible 
to  graduate  from  NPS. 

B.  GRADUATION  ELIGIBLE  POPULATION 

The  previous  section  considered  being  able  to  graduate  as  a  definition  of 
student  success.  In  this  section  we  consider  defining  student  success  by 
examining  an  award  that  a  student  can  receive  at  the  time  of  graduation;  “With 
Distinction.”  This  will  be  done  by  analyzing  only  the  subset  of  the  population  that 
is  graduation  eligible  (2479  students). 

1.  “With  Distinction”  as  Response  Variabie 

The  other  possible  binomial  response  variable  for  determining  student 
success  is  whether  or  not  a  student  is  awarded  a  “With  Distinction”  upon 
graduation  (as  voted  on  by  respective  college  faculty).  In  addition  to  the  pay 
grades  E-6,  E-7,  and  0-7  being  suppressed  in  this  model,  we  also  removed  the 
predictor  “Retake”  for  the  same  reason.  None  of  the  suppressed  factors  or 
variables  were  identified  significant  prior  to  their  removal  from  this  model.  The 
results  of  this  model  are  shown  below  in  Table  9.  It  provides  the  “Estimate”  of  the 
coefficients,  the  marginal  effect,  the  standard  error,  z-value,  and  p-value 
(Pr(>|z|))  for  each  predictor  variable. 
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Table  9.  Summaty  results  of  the  binomial  GLM  with  graduation 
eligible  population.  “With  Distinction”  award  as  response 
variable.  Four  variables  have  strong  significance  (after  IRRA, 

2014). 


Coef  £±clent:s : 

Estimate 

Mrg.  Effect 

Std .  Error 

z-value 

PrOlzl) 

Intercept 

139.219 

107 .187 

1.299 

0.164 

DL 

-0.527 

-0.015 

0.265 

-1.989 

0.047 

0-4 

1.021 

0.038 

0.302 

3.387 

0.001 

0-5 

2.279 

0.172 

0.442 

5.160 

2.47e-07 

0-6 

3.523 

0.470 

0.810 

4.351 

1.35e-05 

Aviator 

0.848 

-0.030 

0.369 

2.299 

0.215 

Nuke 

1.115 

0.052 

0.428 

2.605 

0.009 

SOF 

1.468 

0.085 

0.592 

2.480 

0.013 

HR 

-0.551 

-0.013 

0.812 

-0.680 

0.497 

Supply 

0.235 

0.007 

0.536 

0.439 

0.660 

Engineer 

1.248 

0.058 

0.402 

3.107 

0.002 

I  DC 

1.137 

0.051 

0.412 

2.756 

0.006 

Med 

0.309 

0.010 

0.824 

0.375 

0 . 707 

Misc 

-0.553 

-0.012 

1.118 

-0.495 

0.621 

Start  Acad  Year 

-0.071 

0.002 

0.533 

-1.341 

0.180 

since  Ugrad 

-0.064 

-0.002 

0.039 

-1.667 

0.095 

U3NA  Grad 

-0.064 

0.003 

0.198 

0.486 

0.627 

GSEAS 

0.046 

0.018 

0.274 

0.169 

0.866 

3301 S 

0.294 

0.009 

0.252 

1.167 

0.243 

APC  delta 

0.297 

0.009 

0.037 

7.972 

1 . 56e-15 

Refresher 

0 .167 

0.005 

0.294 

0.566 

0.571 

Signif  Codes: 

0  *****  0 . 

001  •***  0.01 

* '  0.05  * . '  0.1 

■  ' 

Null  Dev;  1112.96  on  2110  df 
Residual  Dev:  925.47  on  2089  df 
AIC:  969.47 


Each  analyzed  military  rank  is  strongly  significantly  different  from  0-3,  the 
base  level,  and  the  only  variable  that  has  retained  strong  significance  following 
the  first  two  models  of  this  study  is  ARC  delta.  It  shows  a  0.9  percent  increase  in 
likelihood  of  graduating  with  distinction  for  each  one  point  increase  in  ARC.  The 
marked  decrease  in  significance  of  DL  can  indicate  that  being  a  DL  student  is  not 
as  detrimental  for  this  high  level  of  student  success  with  only  a  1.5  percent 
decrease  in  likelihood  of  the  honor  for  DL  students.  The  results  of  an  Analysis  of 
Variance  (ANOVA)  for  this  model  are  in  Table  10. 
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Table  10.  ANOVA  table  for  graduation  eligible  population  “with 
distinction”  binomial  LM.  Two  predictors  have  strong 
significance  (after  IRRA,  2014). 


Deg  Frdm 

Deviance 

AIC 

LRT 

Pr (Chi) 

NULL 

920.73 

976.73 

DL 

1 

924.92 

978.92 

4.20 

0.041 

Pay  Grade 

8 

953.42 

993.42 

32.69 

6.99e-05 

Community 

10 

945.96 

981.96 

25.23 

0.005 

Start  Acad  Year 

1 

922.76 

976.76 

2.03 

0.154 

Since  Ugrad 

1 

923.17 

977.17 

2.45 

0.118 

USNA  Graduate 

1 

920.90 

974.90 

0.17 

0.679 

School 

2 

922.54 

974.54 

1.82 

0.403 

APC  delta 

1 

991.06 

1045.06 

70.34 

<2.2e-16 

Refresher 

1 

921.01 

975.01 

0.29 

0.593 

Signif  Codes: 

0  '***’  0.001 

'**'  0.01 

'*'  0.05  '. 

0.1  '  ' 

For  the  first  time,  we  see  smaller  significance  for  the  DL  variable  in  the 
ANOVA  table  for  this  model,  interestingly  Navy  officer  Community  shows  good 
significance  in  this  ANOVA  while  no  factors  show  that  level  of  significance 
separately.  ARC  delta  and  Military  Pay  Grade  are,  once  again,  very  significant. 
This  provides  evidence  that  the  “With  Distinction”  award  as  a  very  likely  definition 
of  student  success  for  both  resident  and  DL  students.  Analyzing  a  smaller  subset 
of  the  data,  only  students  that  are  graduation  eligible,  has  provided  more  insight 
into  what  leads  to  student  success. 

2.  Recursive  Partitioning:  “With  Distinction” 

We  discovered  that,  by  using  all  initial  variables,  we  were  unable  to 
produce  a  noncomplex  classification  tree  (CT).  This  is  due  to  the  fact  that  RPs 
can  be  overly  influenced  by  variables  with  many  factors  (Therneau  &  Atkinson, 
2014).  Therefore,  we  removed  “MilitaryPayGrade”  and  “Community”  from  the 
model  to  provide  a  cleaner  CT.  The  results  of  recursive  partitioning  with  “With 
Distinction”  as  the  response  variable  are  shown  below  in  Figure  18.  It  provides 
the  hierarchical  position  of  each  node  and  its  successive  child  nodes,  which 
variable  is  split,  the  number  of  observations  in  that  node,  and  the  probability  of 
graduating  “With  Distinction.” 
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YES 

APC  Delta  >  4 

N=2479  Prob=6.8% 

NO 

Years  Since  Un 
N=973 

dergrad  Degree  <  14?  N=1506 

Prob=87.9%  Prob=3.4% 

USNA  Gi 
N=779 

N=1 

rad  u  ate?  Prob=; 

Prob=90.0% 

194 

20.6% 

N=60  N=134 

Prob=87.3%  Prob=38.3% 

Figure  18.  RP  classification  tree  with  graduation  eligible  population.  “With 
Distinction”  as  response  variable.  Navy  officers  With  an 
APCdelta  of  4  or  better,  who  graduated  from  the  USNA  less  than 
14  years  have  an  87.3%  probability  of  graduating  “with 
distinction”  (after  IRRA,  2014). 


This  is  the  first  RP  to  identify  APC  delta  as  an  important  predictor  and  it  is 
the  best  predictor  for  the  first  split.  This  indicates  that  an  APC  delta  greater  than 
or  equal  to  4  vastly  improves  ones  opportunity  to  graduate  “With  Distinction,” 
from  3.4%  to  87.9%.  After  that,  time  since  graduation  (less  than  or  equal  to  14 
years,  which  correlates  with  military  pay  grade)  and  USNA  graduate  status  are 
important.  This  RP  indicates  that  a  student  with  an  APC  that  is  4  or  better,  who 
graduated  from  the  Naval  Academy  less  than  14  years  prior  to  starting  studies  at 
NPS  has  the  best  chance  to  graduate  with  distinction  from  NPS.  What  is  most 
important  in  this  RP  is  the  absence  of  DL.  This  indicates  that  DL  is  not  an 
important  predictor  when  determining  this  high  level  of  student  performance. 


48 


C.  DISENROLLED  STUDENTS 

To  best  ensure  student  success,  it  is  advisable  to  develop  a  better 
understanding  of  students  who  become  disenrolled  from  studies  at  NPS  (be  it  for 
academic  or  administrative  reasons).  For  this  purpose  I  returned  to  analyzing  the 
total  population  (2,633  students). 

1.  Disenrolled  as  Response  Variable 

The  data  provides  an  opportunity  to  also  look  at  the  opposite  of  student 
success:  disenrollment  from  NPS.  Running  a  model  with  this  variable  as  a 
binomial  response  provides  insight  into  which  predictors  may  lead  to  a  student 
failure.  We  again  suppressed  the  military  pay  grades  of  E-6,  E-7,  and  0-7  for  the 
same  reason  as  before.  We  did,  however,  bring  the  variable  “Retake”  back  into 
this  model.  The  results  of  this  model  are  shown  below  in  Table  11 .  It  provides  the 
estimate  of  the  coefficient,  the  marginal  effect,  the  standard  error,  z-value,  and  p- 
value  (Pr(>|z|))  for  each  predictor  variable. 
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Table  1 1 .  Summary  Results  of  the  binomial  GLM  with  total  population. 
“Disenrolled”  as  response  variable.  Four  variables  have  strong 
significance  (after  IRRA,  2014). 


Coefficient's : 

Estimate 

Mrq.  Effect 

Std .  Elrror 

z -value 

Pr (> 1 z 1 ) 

Intercept 

195.902 

90.694 

2.160 

0.031 

DL 

2.301 

0.115 

0.286 

8.046 

8.58e-16 

0-1 

1.193 

0.076 

0.638 

1.871 

0.061 

0-2 

-0.376 

-0.011 

0.584 

-0.644 

0.519 

0-4 

-0.277 

-0.009 

0.282 

-0.980 

0.327 

0-5 

-0.874 

-0.023 

0.413 

-0.211 

0.035 

0-6 

-2.752 

-0.350 

0.968 

-2.842 

0.004 

Aviator 

-1.373 

-0.042 

0.267 

-5.146 

2.67e-07 

Nuke 

0.211 

0.008 

0.340 

0.620 

0.535 

SOF 

-0.986 

-0.023 

1.115 

-0.844 

0.377 

HR 

-0.666 

-0.018 

0.505 

-1.321 

0.187 

Supply 

-1.665 

-0.034 

0.657 

-2.536 

0.011 

Engineer 

-0.730 

-0.021 

0.380 

-1.919 

0.055 

I  DC 

-0.421 

-0.013 

0.346 

.1.219 

0.223 

Med 

-1.007 

-0.024 

0.4  91 

-2.051 

0.040 

Misc 

0.331 

0.013 

0.495 

0.668 

0.504 

LDO 

0.434 

0.019 

0.820 

0.530 

0.596 

Start  Acad  Year 

-0.100 

-0.004 

0.045 

-2.205 

0.027 

Since  Ugrad 

0.105 

0.004 

0.030 

3.496 

0.001 

OSNA  Grad 

-0.015 

-0.001 

0.206 

-0.073 

0.942 

GSEAS 

0.570 

0.023 

0.251 

2.268 

0.023 

GSOIS 

0.669 

0.028 

0.245 

2.731 

0.006 

APC  delta 

-0.095 

-0.003 

0.031 

-3.088 

0.002 

Refresher 

-0.513 

-0.016 

0.437 

-1.175 

0.240 

Retake 

2.514 

0.280 

0.450 

5.584 

2.35e-08 

Signif  Codes: 

0  ' ***  '  0 . 

001  '**'  0.01  ’* 

0.05  ' . ■  0.1 

’  ' 

Null  Dev:  1229.72  on  2248  df 
Residual  Dev:  951.92  on  2224  df 
AIC:  1001.9 


This  model  lends  valuable  insight  into  what  can  lead  to  student  failure  as 
defined  by  disenrollment  from  NPS.  Of  the  strongly  significant  variables,  DL 
students  have  an  11.5  percent  greater  likelihood  of  disenrollment  than  their 
resident  counterparts.  A  one-point  increase  in  ARC  delta  is  associated  with  only 
a  very  slight  0.3  percent  decrease  in  likelihood  of  disenrollment.  GSOIS  students 
are  2.8  percent  more  likely  to  disenroll  and  the  most  alarming  revelation  is  that 
retaking  a  course  increases  the  likelihood  of  disenrollment  by  28  percent.  The 
results  of  an  Analysis  of  Variance  (ANOVA)  for  this  model  are  in  Table  12. 
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Table  12.  ANOVA  table  of  the  total  population.  “Disenrolled”  binomial 
LM.  Four  predictors  have  strong  significance  (after  IRRA, 

2014). 


Deg  Frdm 

Deviance 

AIC 

LRT 

Pr (Chi) 

NULL 

951.92 

1007.9 

DL 

1 

1029.72 

1083.7 

77.80 

<2 .2e-16 

Pay  Grade 

8 

969.32 

1009.3 

17.40 

0.026 

Community 

10 

1002.08 

1038.1 

50.16 

2.494e-07 

Start  Acad  Year 

1 

956.79 

1010.8 

4.87 

0.027 

Since  Ugrad 

1 

964.68 

1018.7 

12.76 

0.000 

USNA  Graduate 

1 

951.92 

1005.9 

0.01 

0.942 

School 

2 

960.87 

1012.9 

8.95 

0.011 

APC  delta 

1 

961.53 

1015.5 

9.61 

0.002 

Refresher 

1 

953.39 

1007.4 

1.47 

0.225 

Retake 

1 

976.45 

1030.4 

24.51 

7.41e-07 

Signif  Codes: 

0  ■***■  0.001 

'**'  0.01 

***  0.05  ' . * 

0.1  '  ' 

We  again  see  strong  significance  for  the  DL  variable  in  this  ANOVA  table. 
Retaking  a  course  also  shows  strong  significance  along  with  time  since 
undergraduate  studies.  Notably,  NPS  school  attended  and  ARC  delta  are  slightly 
less  significant  than  DL  and  “Retake”  in  this  model.  By  analyzing  the  opposite  of 
student  success  we  are  noticing  which  variables  are  solid  indicators  in  this  data 
set. 


2.  Recursive  Partitioning:  Disenrolled  Binomial 

The  results  of  recursive  partitioning  with  “Disenrolled”  as  the  response 
variable  are  shown  below  in  Figure  19.  It  provides  the  hierarchical  position  of 
each  node  and  its  successive  child  nodes,  which  variable  is  split,  the  number  of 
observations  in  that  node,  the  loss  within  that  node,  the  value  of  the  respondent 
for  that  node  and  the  probability  of  being  disenrolled  in  parenthesis. 
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YES 


Distance  Learning? 

N=2633  I  Prob=9.7% 


GSEAS  or  GSOIS? 
N=1187  .  Prob=18.0% 


GSEAS? 

N=454  I  Prob=31.1% 


N=733 

Prob=10.0% 


N=287 


Prob=35.9% 


ARC  delta  <7?  N=167 

Prob=22.8% 


N=274 

Prob=33.6% 


N=13 

Prob=15.3% 


NO 


N=1446 

Prob=2.8% 


Figure  19.  RP  Classification  tree  of  the  total  population.  “Disenrolled”  as 
response  variable.  Probability  of  disenrolling  for  274  Navy  officer 
students  in  GSEAS  DL  programs,  with  APC  delta  less  than  or 
equal  to  seven,  is  33.6%  (after  IRRA,  2014). 


This  RP  continues  the  trend  of  DL  being  one  of  the  best  predictors  for 
determining  response.  After  that  the  trend  continues  by  splitting  the  three  schools 
in  two  steps.  Lastly,  APC  delta  is  identified  as  third  best  predictor  with  a  split  at  7. 
Interpretation  of  this  CT  means  that  a  GSEAS  DL  student  with  an  APC  delta  less 
than  7  has  the  greatest  likelihood  of  being  disenrolled. 

3.  TQPR  as  Response  Variable  (Disenrolled  Student  Population) 

A  necessary  follow  up  to  analyzing  which  students  among  the  total 
population  end  up  disenrolled  from  NPS,  is  to  analyze  the  subset  of  disenrolled 
students  to  see  which  predictors  can  signal  disenrollment  from  NPS.  The  results 
of  this  model  are  shown  below  in  Table  13.  It  provides  the  “Estimate”  of  the 
coefficients,  the  standard  error,  z-value,  and  p-value  (Pr(>|z|))  for  each  predictor 
variable. 
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Table  13.  Summary  results  of  the  simple  linear  regression  of  the 
disenrolled  population.  “TQPR”  as  response  variable.  One 
predictor  has  significance  (after  IRRA,  2014). 


TQPR  of 

Disenrolled 

Response 

Coefficients : 

Estimate 

Std.  Error 

t-value 

PrOltl  ) 

Intercept 

28.718 

111.064 

0.259 

0.796 

DL 

-1.178 

0.404 

-2.912 

0.004 

*  * 

0-1 

-0.604 

0.583 

-1.037 

0.302 

0-2 

0.188 

0.677 

0.278 

0.782 

0-4 

0.580 

0.329 

1.764 

0.080 

. 

0-5 

0.590 

0.481 

1.227 

0.222 

0-6 

3.143 

1.330 

2.362 

0.019 

★ 

Aviator 

0.227 

0.318 

0.713 

0.477 

Nuke 

0.095 

0 . 397 

0.240 

0.810 

SOF 

-0.983 

1.440 

-0.683 

0.496 

HR 

-0.029 

0.610 

-0.048 

0.962 

Supply 

-0.601 

0.870 

-0.683 

0.962 

Engineer 

0.034 

0 . 474 

0.072 

0.943 

I  DC 

0.177 

0.465 

0.380 

0.705 

Med 

0.949 

0.603 

1.076 

0.284 

Misc 

-0.152 

0.599 

-0.254 

0.800 

LDO 

-0.132 

0.783 

-0.169 

0.866 

Start  Acad  Year 

-0.012 

0.055 

-0.220 

0.826 

Since  Ugrad 

-0.042 

0.032 

-1.300 

0.196 

USNA  Grad 

0.218 

0.265 

0.823 

0.412 

GSEAS 

-0.839 

0.339 

-2.477 

0.014 

★ 

GSOIS 

-1 .764 

0.311 

-5.678 

6. 86e-08 

★  ★  ★ 

ARC  delta 

0.040 

0.034 

1.186 

0.237 

Refresher 

0.481 

0 . 596 

0.806 

0.421 

Retake 

-0.118 

0.540 

-0.218 

0.828 

Signif  Codes: 

0  '***•  0.001  '**'  0.01  '* 

'  0.05  ’ . '  0 

.1  ■  ' 

Residual  SE: 

1 

.376  on  150 

df 

Mult  R''2: 

0 

.3281 

Adj  R''2:  0. 

2206 

F-stat : 

3 

.052  on  24 

&  150  df 

p-value:  1. 

815e-10 
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Only  one  predictor  shows  strong  significance  in  this  model,  the  Graduate 
School  of  Operational  and  Information  Sciences  (GSOIS).  With  all  other  variables 
being  equal  a  student  who  was  disenrolled  from  GSOIS  will  have,  on  average,  a 
TQPR  1.76  points  less  than  students  in  the  other  two  graduate  schools.  DL  may 
be  slightly  less  significant,  but  it  cannot  be  overlooked  that  a  disenrolled  DL 
student  has  a  TQPR  that  is  1.18  points  less,  on  average,  than  disenrolled 
resident  students.  The  results  of  an  Analysis  of  Variance  (ANOVA)  for  this  model 
are  in  Table  14. 


Table  14.  ANOVA  table  of  the  disenrolled  population  simple  linear 
regression.  “TQPR”  as  response  variable.  One  predictor  has 
strong  significance  (after  IRRA,  2014). 


Deg  Frdm 

Svnn  Sq 

Mean  Sq 

F  value 

Pr  (>F) 

DL 

1 

19.78 

19.78 

10.45 

0.002 

Pay  Grade 

5 

20.43 

4.09 

2.16 

0.062 

Community 

10 

19.19 

1.92 

2.01 

0.434 

Start  Acad  Year 

1 

3.44 

3.44 

1.82 

0.180 

Since  Ugrad 

1 

8.07 

8.07 

4.26 

0.041 

OSNA  Graduate 

1 

0.04 

0.04 

0.02 

0.884 

School 

2 

31.89 

63.79 

16.85 

2.511e-07 

APC  delta 

1 

2.66 

2.65 

1.40 

0.238 

Refresher 

1 

1.17 

1.17 

0.62 

0.432 

Retake 

1 

0.09 

0.09 

0.05 

0.828 

Residuals 

150 

283.97 

1.89 

Signif  Codes: 

0  ■***■  0.001 

'**■  0.01 

0.05  0.1 

1  f 

This  ANOVA  table  is  consistent  with  the  summary  of  this  model.  School  name  is 
very  significant  and  DL  status  is  significant  at  the  99.9%  level. 
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4.  Recursive  Partitioning  (TQPR  as  Response  Variabie) 

The  results  of  recursive  partitioning  with  the  disenrolled  population  and 
TQPR  as  the  response  variable  are  in  Figure  20.  It  provides  the  hierarchical 
position  of  each  node  and  its  successive  child  nodes,  which  variable  is  split,  the 
number  of  observations  in  that  node,  and  the  average  value  of  the  respondent. 


YES 

GSEAS  or  GSOIS? 

N=255  TQPR=2.33 

NO 

Distance  Learning  Student?  N=80 

N=175  TQPR=1.93  TQPR  =  3  21 

N=141 

N=34 

TQPR  =  1.63 

TQPR  =  3.17 

Figure  20.  RP  regression  tree  of  the  disenrolled  population.  “TQPR”  as 
response  variable.  Average  TQPR  of  141  Navy  officer  students 
in  DL  students  in  GSEAS  and  GSQIS  is  1.63  (after  IRRA,  2014). 


This  is  the  shortest,  most  straightforward  and  telling  of  the  RPs  developed. 
The  first  split  is  between  “GSEAS,  GSQIS”  and  “GSBPP.”  Beneath  “GSEAS, 
GSQIS”  is  DL  status.  141  of  the  255  disenrollees  (56.47  percent)  from  NPS  were 
DL  students  from  these  schools  with  an  average  GQPR  of  1 .63. 
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D.  ANALYSIS  SUMMARY 

This  analysis  produced  twelve  separate  models  to  explore  the  data  and 
determine  if  there  is  a  difference  between  the  performance  of  DL  Navy  officer 
students  and  resident  Navy  officer  students  at  NPS.  In  addition,  we  looked  at 
each  model  to  see  if  the  APC  is  a  valid  predictor  of  student  success.  Table  15  is 
a  summary  of  the  results  of  each  model  with  respect  to  the  two  primary 
predictors  of  interest:  “IsDL”  and  “APCdelta.” 


Table  15.  Summary  of  model  results  with  a  Focus  on  DL  and  APdelta 

(after  IRRA,  2014). 


Model  Response 

Significant? 

(Coefficient  /  Marginal  Effect)*** 

RP  Level 

TQPR 

(N=2,633) 

DL:  Yes  (-0.32) 

APCdelta:  Yes  (+0.03) 

DL;  2 

APCdelta;  N/A 

Graduation  Eligible* 
(N=2,633) 

DL;  Yes  (-05.7%) 
APCdelta:  Yes  (+0.02%) 

DL:  2 

APCdelta;  N/A 

With  Distinction* 
(N=2.479) 

DL;  No 

APCdelta:  Yes  (0.9%) 

DL;  No 
APCdelta;  1 

Disenrolled* 

(N=2,633) 

DL;  Yes  (1 1 .5%) 

APCdelta;  No 

DL;  1 

APCdelta:  4 

Disenrolled  TQPR 
(N=255) 

DL:  No 

APCdelta:  No 

DL:  2 

APCdelta;  N/A 

*logit  link  and  RP  classiTication 
model 

***  >0.001  significance 

For  the  regression  models,  Table  15  shows  whether  a  variable  was 
statistically  significant  at  the  0.001  percent  level,  and  if  so,  the  estimate  of  its 
numeric  (TQPR  response)  or  marginal  probability  effect  (binomial  response).  For 
the  RP  models,  we  provide  the  level  of  the  highest  split  for  that  variable.  Level 
one  means  that  the  listed  variable  is  featured  as  the  root  of  its  respective  tree. 
Level  two  means  that  variable  is  identified  as  the  level  immediately  below  the 
root,  and  so  on.  When  ‘N/A’  is  listed,  that  variable  is  not  identified  as  important 
enough  to  be  identified  by  the  RP  model. 
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V.  CONCLUSION 


A.  ANSWERS  TO  THE  RESEARCH  QUESTIONS 

1 .  Question  1 :  Is  the  NFS  AFC  a  Valid  Fredictor  of  Student 
Success  in  Both  DL  and  Resident  Frograms? 

The  predictor  “APCdelta”  is  identified  as  having  very  strong  significance  in 
determining  a  student’s  TQPR  (in  the  simple  linear  model)  and  whether  or  not  a 
student  is  eligible  for  graduation.  “APCdelta”  also  has  very  strong  significance  in 
whether  or  not  a  student  graduates  with  distinction  and  is  slightly  less  significant 
in  determining  if  a  student  is  disenrolled  in  those  two  binary  response  general 
linear  models. 

In  regards  to  the  recursive  partition  models,  an  “APCdelta”  greater  than  or 
equal  to  four  is  identified  as  the  best  determinant  of  whether  or  not  a  student 
graduates  with  distinction.  Conversely,  “APCdelta”  less  than  or  equal  to  seven  is 
identified  as  the  fourth  most  important  predictor,  and  therefore  having  some 
association,  of  whether  or  not  a  student  is  disenrolled  from  NPS  for  either 
academic  or  administrative  purposes. 

These  two  recognitions  by  the  RP  model  (an  APC  delta  of  four  or  better 
can  lead  to  great  success  while  one  of  seven  or  lower  can  also  lead  to 
disenrollment)  make  claiming  the  APC  as  a  valid  predictor  of  student  success  a 
little  difficult.  All  three  of  the  APC  digits  were  considered  of  equal  weight  in  this 
study.  The  “gray  area”  (APC  delta  scores  from  four  to  seven)  indicate  the 
necessity  for  a  follow-up  study  where  each  APC  digit  is  examined  separately  to 
determine  which  has  more  effect  on  student  performance.  Given  that  the 
identification  of  an  APC  of  four  or  better  ranks  at  the  top  of  its  RP  model  for 
graduating  with  distinction  and  identification  of  an  APC  of  seven  or  lower  ranks 
fourth  in  predicting  disenrollment  we  have  determined  that,  as  a  whole,  the  APC 
is  a  valid  predictor  of  student  success. 
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2.  Question  2:  Do  Graduate  Students  Achieve  a  Higher  Levei  of 
Student  Performance  in  a  Resident  Education  or  in  a  Distance 
Learning  Education? 

The  “IsDL”  predictor  is  identified  as  having  strong  significance  in  all  but 
two  of  the  Generalized  Linear  Models.  The  other  two  LMs  (“With  Distinction” 
binomial  and  TQPR  of  the  disenrolled  population)  report  “IsDL”  having  slightly 
less  significance  (at  the  00.1%  level).  In  each  model,  the  direction  of  correlation 
is  negative  when  determining  TQPR,  graduation  eligibility,  and  graduating  with 
distinction.  The  direction  of  correlation  is  positive  for  both  disenrolled  LMs.  This 
does  not  look  good  for  the  DL  program. 

The  recursive  partitioning  models  shed  some  light  on  which  DL  students 
do  not  perform  as  well  as  their  resident  counterparts.  “IsDL:  Y”  is  the  most 
important  predictor  in  determining  if  a  student  is  disenrolled.  It  is  the  second  most 
import  predictor  in  determining  TQPR,  graduation  eligibility,  and  the  TQPR  of  the 
disenrolled  population.  As  previously  mentioned,  RP  models  are  useful  for  finding 
where  interactions  exist  without  having  to  manually  set  up  the  interactions  in  a 
LM.  In  all  but  one  of  the  RPs  that  identify  “IsDL”  as  important,  it  is  followed 
immediately  by  the  school  attended.  The  school  name  precedes  “IsDL”  in  the  RP 
of  the  TQPR  of  the  disenrolled  population. 

This  leads  to  a  mixed  answer  for  this  research  question.  Graduate 
students  in  the  Graduate  School  of  Business  and  Public  Policy  perform  equally 
well  in  DL  and  resident  programs.  DL  graduate  students  in  the  Graduate  School 
of  Engineering  and  Applied  Sciences  do  not  perform  as  well  as  their  resident 
counterparts.  The  greatest  disparity  between  performance  of  DL  and  resident 
students  is  in  the  Graduate  School  of  Qperational  and  Information  Sciences. 

3.  Question  3:  What  Student  Attributes  Lead  to  Success  in 
Distance  Learning  versus  Resident  Learning  (and  Vice  Versa) 
and  Where  Do  they  Differ? 

This  study  researched  two  different  definitions  of  student  success; 
achieving  eligibility  to  graduate,  and  graduating  with  distinction.  We  also  studied 
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the  opposite  of  student  success  by  analyzing  what  could  lead  to  disenrollment 
from  NPS. 


a.  Graduation  Eligible 

The  predictors  and  factors  listed  in  Table  15  provide  the  answer  on  which 
attributes  aid  in  reaching  the  basic  standard  of  student  success  at  NPS.  The  LM 
model  reveals  that  Aviation  officers  do  particularly  well  at  NPS  and  it  is  best  to 
have  a  high  APC  delta.  It  also  shows  that  DL  students,  GSOIS  students  and 
students  that  had  to  retake  a  course  are  less  likely  to  have  a  graduation  eligible 
GQPR.  The  RP  adds  to  the  LM  by  identifying  0-3  to  0-5  Submarine  Officers  and 
Limited  Duty  Officers  taking  DL  courses  through  the  GSEAS  or  the  GSOIS  have 
had  difficulty  maintaining  a  graduation  eligible  GQPR. 

b.  Graduating  “with  Distinction” 

Table  5  provides  a  summary  of  the  student  attributes  that  can  lead  to 
graduating  with  distinction.  Remarkably  these  are  the  only  models  not  to  identify 
DL  status  as  strongly  significant.  The  LM  is  very  straightforward;  students  of  pay 
grade  0-4  through  0-6  and  those  with  a  high  APC  delta  are  most  likely  to 
graduate  with  distinction.  The  RP  model  is  the  most  complex  of  all  twelve.  It 
agrees  with  the  LM  by  identifying  the  APC  delta  and  higher  paygrades  as  the 
most  important  predictors  and  then  goes  on  to  list  USNA  graduates  and  a 
handful  of  officer  communities. 

c.  Disenroiied 

Lastly,  the  summary  of  the  models  used  to  analyze  student  failure  is  listed 
in  Table  15.  The  LM  reveals  that  being  a  DL  student,  greater  time  since 
undergraduate  degree  and  having  to  retake  a  course  as  increasing  the  likelihood 
of  disenrollment  from  NPS  and,  if  disenroiied,  it  is  very  likely  the  student  was 
taking  classes  through  the  GSOIS.  The  RP  models  are  consistent  with  the  LMs; 
however  it  is  interesting  to  see  that  an  APC  delta  greater  than  or  equal  to  7  is 
identified  by  the  RP  for  which  DL  students  become  disenroiied  from  the  GSOIS. 
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A  return  look  at  the  summary  for  this  RP  reveals  that  the  classification  tree 
identified  this  group  as  only  2  of  the  255  (0.8%)  Navy  officer  students  in  the  data 
that  were  disenrolled  from  NPS. 

B.  POSSIBLE  FOLLOW-UP  WORK 

This  is  the  first  study  to  analyze  the  performance  of  distance  learning 
Navy  officer  students  and  their  resident  counterparts  at  NPS  by  looking  at  the 
population’s  course  grades  and  academic  awards  received  as  definitions  of 
student  success.  The  only  data  analyzed  was  academic  and  career  related 
information  provided  by  the  NPS  Institutional  Research,  Reporting  and  Analysis 
office.  Biographical  data  such  as  age,  race,  gender,  etc.,  was  not  analyzed; 
therefore  no  inference  can  be  made  regarding  those  traits  and  academic 
achievement  at  NPS. 

This  study  was  a  high-level  overview  of  Navy  officer  academic 
performance  at  NPS.  Follow  up  work  more  closely  analyzing  each  digit  the  APC 
is  recommended  especially  given  the  fact  that  the  standards  for  APC2  and  for 
APC3  were  adjusted  in  2010  and  2012,  respectively.  Another  more  in-depth 
opportunity  to  analyze  this  data  is  to  study  student  performance  in  curricula 
within  the  NPS  schools. 

The  original  data  contained  academic  information  on  all  DL  students  to 
attend  NPS  since  2006  and  all  resident  students  to  attend  NPS  since  2008.  A 
similar  study  is  possible  for  analysts  interested  in  other  military  officer 
populations.  Department  of  Defense  civilian  students  and  all  international 
students.  For  those  researchers  willing  to  take  on  the  responsibility  of  handling 
Personally  Identifiable  Information  (Pll),  it  is  possible  to  use  this  data  to  link  each 
observation  to  the  actual  student,  by  working  with  the  Defense  Manpower  Data 
Center  (DMDC),  and  analyze  student  career  performance  after  graduating  from 
NPS. 
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C.  RECOMMENDATIONS 

Success  at  NPS  is  a  very  achievable  goal  for  any  Navy  officer  with  the 
desire  to  continue  his  or  her  education  and  improve  his  or  her  value  to  the  United 
States  Navy.  Of  the  2,974  Navy  officers  to  attend  NPS  since  2008  and  the  take 
classes  via  distance  learning  since  2006,  only  268  were  disenrolled  due  to 
academic  or  administrative  purposes. 

The  Academic  Profile  Code  is  a  valid  indicator  of  potential  student 
success.  Graduating  from  NPS  is  a  good  definition  of  academic  success,  but  the 
better  definition  of  a  higher  level  of  success  is  graduating  with  distinction 
because  it  is  the  only  model  not  to  identify  DL  status  and  school  attended  as  a 
strong  predictor. 

With  an  better  than  90%  level  of  success,  and  no  real  distinguishable 
characteristic  that  leads  to  success,  it  is  more  efficient  to  identify  the  predictors  of 
student  failure  and  determine  where  NPS  can  make  adjustments  to  best  ensure 
the  success  of  its  students.  There  is  a  subset  of  the  student  population  that  this 
study  has  identified  through  valid  statistical  exploration  of  the  data.  There  is  a 
common  theme  among  eleven  of  the  twelve  of  the  models  developed  in  this 
study:  DL  students  in  the  two  technical  schools  (GSEAS  and  GSOIS)  perform  at 
a  level  below  that  of  their  resident  student  counterparts. 

It  is  too  early  to  determine  the  exact  reason  for  this  population’s  lack  of 
student  success.  One  possible  explanation  is  that  the  students  obligate  to  a  DL 
program  without  a  full  understanding  of  the  time  commitment  required,  beyond 
their  full-time  jobs,  for  success.  It  may  also  be  possible  that  students  and 
instructors  are  fully  prepared  for  the  DL  experience,  but  the  high  level  of 
comprehension  required  for  the  technical  curriculum  offered  by  both  GSEAS  and 
GSOIS  make  it  very  difficult  for  the  knowledge  to  be  transferred  effectively  via  the 
DL  medium. 

One  possible  solution  is  to  offer  a  “refresher”  quarter  to  prospective 
students  interested  in  DL  programs  in  either  GSEAS  or  GSOIS.  This  refresher 
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quarter  can  be  offered  on  a  “no  commitment  to  full  enrollment”  basis.  This  way, 
Navy  officers  that  have  been  out  of  school  for  some  time  have  the  opportunity  to 
gauge  their  academic  readiness,  determine  if  the  study  time  required  fits  into 
their  lifestyle,  and  get  a  feel  for  the  level  of  the  academic  rigor  inherent  those 
programs.  This  refresher  quarter  can  also  be  beneficial  to  the  instructors  to 
develop  teaching  skills  useful  for  the  DL  model. 
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